Our approach is lossless.
Original titleOur approach is lossless. The same tokens visit the same experts. The architecture, routing decisions and training objective stay unchang...
AISummary
The same tokens visit the same experts. The architecture, routing decisions and training objective stay unchanged. The model computes the same function. By organizing where experts and tokens live we reduce the communication needed to do the same work.
Source: Zyphra · x.com