Skip to content
View original post on X: Ai2Official
Published
34/100

Ai2 tests whether hybrid architectures beat transformers on state tracking

AIAi2 trained LLMs directly on StateBench, then evaluated them on longer inputs to separate architecture effects from pretraining effects. Hybrid models with Gated DeltaNets were near-perfect on the first three levels, while transformers degraded as input length grew and could not solve general finite-state tasks.
Ai2Verified on XOfficial
@allen_ai
View on X

Part of a thread · earlier post

To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs.

Hybrids + Gated DeltaNets were near-perfect on the first 3 levels; transformers degraded with length & couldn’t solve general finite.

Source details

Ai2 · x.comPublished · added here

Original title: To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs.

Topics

ReasoningData & trainingPapers & researchEvals & benchmarks