Finding a passage in a long book? Look for the right chapter, then section, rather than rereading every sentence.
ZHA applies the same idea to the KV cache. We recursively summarize chunks into a tree and route top-down. If a node is skipped, all keys under it aren’t scored.