How fast could it be?
Our GPU roofline model projects ZHA overtaking dense attention at ~48K tokens and MSA at ~480K. At 1B tokens, it projects ~373–417× speedup vs dense and ~52–58× vs MSA.
Zyphra projects ZHA speedups over dense attention and MSA at 1B tokens
AIZyphra's GPU roofline model projects its ZHA attention overtaking dense attention at about 48K tokens and MSA at about 480K tokens. At 1 billion tokens, the model projects a 373–417x speedup over dense attention and 52–58x over MSA.