Step 5 Preview's sparse attention starts with region summaries.
The indexer scores summaries, selects regions, then attends to the selected tokens. Unfinished regions stay in the local window.
The backbone combines sliding-window and sparse attention with MoE and multi-token prediction, alongside the vision processing path.
2/6
vLLM describes sparse attention in Step 5 Preview model
AISummary
vLLM says Step 5 Preview's sparse attention first builds region summaries, then an indexer scores them and selects regions to attend to. Unfinished regions remain in the local window. The backbone combines sliding-window and sparse attention with MoE and multi-token prediction, alongside a vision processing path.
Post on XView on X
vLLMVerified on X
@vllm_project
Part of a thread · earlier post
Source: vLLM · x.comPublished