vLLM breaks down Muse Spark Open's latent MoE and hybrid attention design

Inside Muse Spark Open: latent MoE and hybrid attention.

Learned pooler and unpooler projections take the 8,192-wide hidden state down to 4,096 for expert computation, then restore it. All 62 decoder layers use MoE, with 256 experts per layer.

46 layers use a 2,048-token sliding window; 16 attend to the full context. Image and video inputs use a Perception Encoder, with a separate 118M audio encoder.

2/6