Skip to content
View original post on X: Ai2Official
Published
34/100

Ai2 finds frontier model accuracy drops to 13% on longest tasks

AIAi2 tested open pretrained models and frontier models, including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. The frontier models performed well on simpler problems, but accuracy fell sharply as tasks grew harder and inputs longer, reaching 13% or lower on the longest general finite problems.
Ai2Verified on XOfficial
@allen_ai
View on X

Part of a thread · earlier post

We tested open pretrained models + frontier models incl. Gemini 3.8 Flash, Claude Opus 5.5, & GPT-6 Sol + Luna.

Frontier models did well on simpler problems. But as tasks got harder & inputs longer, accuracy dropped sharply—to 13% or lower on the longest general finite problems.

Source details

Ai2 · x.comPublished · added here

Original title: We tested open pretrained models + frontier models incl. Gemini 3.8 Flash, Claude Opus 5.5, & GPT-6 Sol + Luna.

Topics

OpenAI / ChatGPTAnthropic / ClaudeGoogle / GeminiPapers & researchEvals & benchmarks