# Zyphra proposes hierarchical attention for 100M+ token sequences

> Original title: At 100M+ tokens, retrieval may need to grow with the predictive information in a sequence, but much more slowly than the full context.

- Source: Zyphra
- Published: 2026-11-02T22:04:57.000Z
- Bad Influence: https://badinfluence.news/items/ziewwy4pums5kzpnrytqmpknj
- Original: https://x.com/ZyphraAI/status/2117377450384773287

## Summary

Zyphra says retrieval at 100M+ tokens may need to scale with the predictive information in a sequence rather than its full length. It presents hierarchical methods such as ZHA as a way to strike that balance, with a paper on arXiv and a blog post.

## Post

At 100M+ tokens, retrieval may need to grow with the predictive information in a sequence, but much more slowly than the full context.

Hierarchical methods like ZHA offer a path toward this balance.

Paper:  https://arxiv.org/abs/2610.42112
Blog: https://zyphra.com/our-work/zyphra-hierarchical-attention