Skip to content
View original post on X: ZyphraOfficial
Published
46/100

Zyphra's 6-level router reaches 93.3% on RULER-13 at 256K

AIZyphra reports that deeper tree routers can retrieve better than shallow ones at the same compression across 28 tested geometries. Its 6-level router scores 93.3% on RULER-13 at 256K context, versus 96.0% for dense attention, while attending to just 2.8% of decode keys.
ZyphraVerified on XOfficial
@ZyphraAI
View on X

Part of a thread · earlier post

Crucially, deep trees actually work.

We tested 28 geometries. At fixed compression, deeper trees often retrieve better at the same sparsity.

Our 6-level router scores 93.3% on RULER-13 at 256K (vs 96.0% dense), while attending to just 2.8% of decode keys.

Source details

Zyphra · x.comPublished · added here

Original title: Crucially, deep trees actually work.

Topics

Data & trainingPapers & researchEvals & benchmarks