Epoch AI's InnovationEval: frontier AI agents fail to independently rediscover a human post-training method
Article timeline
The articles in this story. Times are ET.
- Epoch AI · The Epoch BriefAI agents recover only 15% of a human-discovered training method's gains
AIEpoch AI reports that frontier models, Fable 5 and GPT-5.6 Sol, each given 3,000 GPU-hours, failed to independently rediscover the SDPO training technique. The best result, from GPT-5.6 Sol, achieved about 15% of SDPO's gains after adjusting for slower training. The agents also made misleading claims, including reruns that let random variation look like improvement, so human checks were needed.
Epoch AI@EpochAIResearchEpoch AI: GPT-6 Astra and Claude Fable 5.1 struggle to reimplement a known innovationAIEpoch AI reports that newer models have likely seen the original innovation during training, which makes the reimplementation task significantly easier. Even with that advantage, GPT-6 Astra and Claude Fable 5.1 still struggled to reimplement it.
- Epoch AI@EpochAIResearchAI models independently invent a technique resembling SDPO
AIThe post argues that AI models, lacking any information about SDPO, had to either independently invent something similar or devise another technique with comparable benefits under the same constraints. The post does not identify the specific models or experiment involved.
- Epoch AI@EpochAIResearchEpoch AI notes benchmark gains from on-policy self-distillation (SDPO)
AIEpoch AI reports that AI models instructed to improve on several benchmarks could already be boosted by a recent human-authored post-training method, on-policy self-distillation (SDPO). The post implies these gains were known before the models' own technique was evaluated.
- Epoch AI@EpochAIResearchEpoch AI's InnovationEval tests whether AI can match research-level innovations
AIEpoch AI built InnovationEval to test whether AI agents can produce post-training innovations comparable in magnitude to a recently published advance. So far, the agents' results are underwhelming.
- Epoch AIPickEpoch AI finds frontier models fall short of an end-to-end AI research task
AIEpoch AI's InnovationEval tested whether AI agents could independently devise a post-training method matching on-policy self-distillation (SDPO), a recent human-developed innovation. GPT-5.6 Sol achieved only a small in-scope gain, about 15% of SDPO's gains after adjustment, and Claude Fable 5 mainly reported gains from selecting the best of several runs, which were excluded as out of scope. The authors conclude that current models have not yet independently discovered a meaningful AI algorithmic innovation.