Sparse attention fixes memory bandwidth, not memory capacity.
“The whole idea of HiCache is trying to offload KV cache from the HBM to a lower tier, higher capacity but lower bandwidth memory, and only load the most relevant set of tokens during computation.”
“Sparse attention doesn’t mitigate the memory capacity bottleneck. It’s a technique that helps a lot with memory bandwidth.”
“During the sparse attention computation, you still need to select the tokens. So you still have to keep all the tokens in context, and you cannot discard them. At some point, it still has to be loaded into HBM.”
“That’s where HiSparse becomes useful, because it can apply the typical caching rules so that you could load the most relevant parts into HBM.”