Zyphra argues ultra-long contexts need subquadratic hierarchical attention
Article timeline
The articles in this story. Times are ET.
ZyphraOfficialZyphra suggests ultra-long contexts may need subquadratic attentionAIZyphra says ultra-long contexts may require subquadratic attention, arguing that attending to everything buries the signal in noise while overly sparse attention loses it. The company suggests hierarchical attention may be essential for attention to work at all, not just for efficiency.
- ZyphraOfficialZyphra proposes hierarchical attention for 100M+ token sequences
AIZyphra says retrieval at 100M+ tokens may need to scale with the predictive information in a sequence rather than its full length. It presents hierarchical methods such as ZHA as a way to strike that balance, with a paper on arXiv and a blog post.
- ZyphraOfficialZyphra explains why top-p attention selection may be subquadratic
AIZyphra says the top-p set in dense attention models scales like a sublinear power law, suggesting attention is already intrinsically sparse. It says its ZHA method uncovers this structure, and the post asks whether top-p selection is actually subquadratic.
