# Ai2 tests whether hybrid architectures beat transformers on state tracking

> Original title: To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs.

- Source: Ai2
- Published: 2026-11-03T17:28:38.000Z
- Bad Influence: https://badinfluence.news/items/q1juuu9ar2xk0uidhs96vr52t
- Original: https://x.com/allen_ai/status/2117670299298599001

## Summary

Ai2 trained LLMs directly on StateBench, then evaluated them on longer inputs to separate architecture effects from pretraining effects. Hybrid models with Gated DeltaNets were near-perfect on the first three levels, while transformers degraded as input length grew and could not solve general finite-state tasks.

## Post

To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs.

Hybrids + Gated DeltaNets were near-perfect on the first 3 levels; transformers degraded with length & couldn’t solve general finite.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWN54LfWEAAAS96.jpg&mode=thumb&exp=1793923200&sig=2a84ddf9d30ec3bb)