Ai2 and Hugging Face release StateBench, a long-document state-tracking benchmark
Article timeline
The articles in this story. Times are ET.
Ai2OfficialAi2 releases StateBench, a dataset for testing state tracking in modelsAIAi2 releases StateBench, a benchmark dataset for researchers to test new models and build better state-tracking training data. The dataset is available on Hugging Face at allenai/state-bench, with more detail in an Ai2 blog post.
- Ai2OfficialAi2 finds frontier model accuracy drops to 13% on longest tasks
AIAi2 tested open pretrained models and frontier models, including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. The frontier models performed well on simpler problems, but accuracy fell sharply as tasks grew harder and inputs longer, reaching 13% or lower on the longest general finite problems.
- Ai2OfficialStateBench splits state tracking into four difficulty levels
AIAi2 introduces StateBench, which divides state tracking into four levels: r-trivial, aperiodic, soluble, and general finite. The levels range from simple tasks such as remembering a fixed fact like a birth date to harder ones such as tracking a ball as cups are repeatedly swapped.
- Ai2OfficialAi2's StateBench finds frontier LLMs drop to 13% accuracy on long state-tracking tasks
AIAi2 releases StateBench, an evaluation of whether LLMs can track state changes over long texts, such as an object changing hands or a variable being reassigned. Ai2 reports that frontier models, including GPT-6 Sol and Claude Opus 5.5, fall to 13% accuracy or lower on the hardest and longest problems. The benchmark is available on GitHub.
- Hugging Face BlogOfficialStateBench tests how language models track state changes over long documents
AIHugging Face introduces StateBench, a benchmark for long-context state tracking across four difficulty levels, from immutable assignment to assignment by reference. In its tests, recurrent and hybrid models generalize much farther than transformers when trained directly on these tasks. The report says frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna still break down on harder forms of state tracking as inputs grow longer.
- Ai2 (Allen Institute for AI)OfficialStateBench tests how language models track state changes over long documents
AIAi2 introduces StateBench, a benchmark that tests how well language models track entity state changes in long documents across four difficulty levels. The team reports that recurrent and hybrid models generalize much farther than transformers when trained directly on these tasks. It also finds that frontier models, including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, still break down on harder state tracking as inputs grow longer.