Routing is also predictable across layers: the experts a token used in one layer tell us which it will likely need next.
Original titleRouting is also predictable across layers: the experts a token used in one layer tell us which it will likely need next. Token shuffling ...
AISummary
Token shuffling moves each token to the GPU holding those experts, inside a transfer that already runs after attention, so it adds no network traffic.
Source: Zyphra · x.com