# Ai2's StateBench finds frontier LLMs drop to 13% accuracy on long state-tracking tasks

> Original title: Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment?

- Source: Ai2
- Published: 2026-11-03T17:28:37.000Z
- Bad Influence: https://badinfluence.news/items/qkagvh0007d63nxmb5crlyeco
- Original: https://x.com/allen_ai/status/2117670295339171994

## Summary

Ai2 releases StateBench, an evaluation of whether LLMs can track state changes over long texts, such as an object changing hands or a variable being reassigned. Ai2 reports that frontier models, including GPT-6 Sol and Claude Opus 5.5, fall to 13% accuracy or lower on the hardest and longest problems. The benchmark is available on GitHub.

## Post

Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment?

On our new StateBench eval, frontier models including GPT-6 Sol & Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems. 🧵

https://github.com/allenai/state-bench

[Video](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fext_tw_video_thumb%2F2117670256801873920%2Fpu%2Fimg%2FngWTR58nsSm8VVSD.jpg&mode=thumb&exp=1793923200&sig=779db1d47a9fa05d)