Zyphra projects ZHA speedup over dense attention and MSA via GPU roofline model
Article timeline
The articles in this story. Times are ET.
ZyphraOfficialZyphra projects ZHA speedups over dense attention and MSA at 1B tokensAIZyphra's GPU roofline model projects its ZHA attention overtaking dense attention at about 48K tokens and MSA at about 480K tokens. At 1 billion tokens, the model projects a 373–417x speedup over dense attention and 52–58x over MSA.
- ZyphraOfficialZyphra's ZHA matches dense models on SWE-bench and beats them on LongBench v2
AIZyphra says its ZHA model is tested on AIME 2026, SWE-bench Verified, Terminal-Bench 2.1, GPQA, RULER, and LongBench v2. Deep ZHA matches a dense model on SWE-bench and outperforms it on LongBench v2 while attending to about 1.1% of decode keys.
- ZyphraOfficialZyphra's ZHA makes attention search sparse with a tree structure
AIZyphra says its ZHA method makes the attention search itself sparse, not just the final attention step. Many sparse-attention methods still pay quadratic cost in their lightweight indexers, which ZHA avoids. Unlike fixed-state models, ZHA keeps the full KV cache for precise retrieval.
- ZyphraOfficialZyphra introduces Hierarchical Attention, a subquadratic attention method
AIZyphra introduces Zyphra Hierarchical Attention (ZHA), which searches a learned tree of compressed summaries to approach subquadratic attention. The company projects speedups of 35× over dense attention and 5× over MSA at 10M tokens, rising to about 400× and 55× at 1B tokens.
- ZyphraOfficialZyphra's ZHA routes KV cache lookups through a summary tree
AIZyphra introduces ZHA, which applies a hierarchical search to the KV cache. It recursively summarizes chunks of keys into a tree and routes queries top-down. When a node is skipped, all keys beneath it go unscored.
