# Zyphra projects ZHA speedups over dense attention and MSA at 1B tokens

> Original title: How fast could it be?

- Source: Zyphra
- Published: 2026-11-02T22:04:55.000Z
- Bad Influence: https://badinfluence.news/items/f7oiuumeewjmbudnva2wgh44v
- Original: https://x.com/ZyphraAI/status/2117377442583363790

## Summary

Zyphra's GPU roofline model projects its ZHA attention overtaking dense attention at about 48K tokens and MSA at about 480K tokens. At 1 billion tokens, the model projects a 373–417x speedup over dense attention and 52–58x over MSA.

## Post

How fast could it be?

Our GPU roofline model projects ZHA overtaking dense attention at ~48K tokens and MSA at ~480K. At 1B tokens, it projects ~373–417× speedup vs dense and ~52–58× vs MSA.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWJsPRLbMAAHWgb.jpg&mode=thumb&exp=1793923200&sig=9a1f9f837a431242)