Skip to content
View original post on X: ZyphraOfficial
Published
26/100

Zyphra proposes hierarchical attention for 100M+ token sequences

AIZyphra says retrieval at 100M+ tokens may need to scale with the predictive information in a sequence rather than its full length. It presents hierarchical methods such as ZHA as a way to strike that balance, with a paper on arXiv and a blog post.
ZyphraVerified on XOfficial
@ZyphraAI
View on X

Part of a thread · earlier post

At 100M+ tokens, retrieval may need to grow with the predictive information in a sequence, but much more slowly than the full context.

Hierarchical methods like ZHA offer a path toward this balance.

Paper: https://arxiv.org/abs/2610.42112
Blog: https://zyphra.com/our-work/zyphra-hierarchical-attention

Source details

Zyphra · x.comPublished · added here

Original title: At 100M+ tokens, retrieval may need to grow with the predictive information in a sequence, but much more slowly than the full context.

Topics

ReasoningData & trainingPapers & research