Skip to content
View original post on X: ZyphraOfficial
Published
34/100

Zyphra introduces Hierarchical Attention, a subquadratic attention method

AIZyphra introduces Zyphra Hierarchical Attention (ZHA), which searches a learned tree of compressed summaries to approach subquadratic attention. The company projects speedups of 35× over dense attention and 5× over MSA at 10M tokens, rising to about 400× and 55× at 1B tokens.
ZyphraVerified on XOfficial
@ZyphraAI
View on X

Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention.

ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵

Source details

Zyphra · x.comPublished · added here

Original title: Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention.

Topics

ReasoningDeployment & engineeringPapers & research