Ai2's StateBench finds frontier LLMs drop to 13% accuracy on long state-tracking tasks

AIAi2 releases StateBench, an evaluation of whether LLMs can track state changes over long texts, such as an object changing hands or a variable being reassigned. Ai2 reports that frontier models, including GPT-6 Sol and Claude Opus 5.5, fall to 13% accuracy or lower on the hardest and longest problems.

Ai2Official

@allen_ai

View on X

Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment?

On our new StateBench eval, frontier models including GPT-6 Sol & Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems. 🧵

https://github.com/allenai/state-bench

Open source

Source details

Ai2 · x.comPublished Nov 3, 12:28 PM ET · added here 12:43 PM ET

Original title: Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment?

Topics

Reasoning Open-source ecosystem Papers & research Evals & benchmarks

Ai2's StateBench finds frontier LLMs drop to 13% accuracy on long state-tracking tasks

Topics

Tags