GPT-6-Astra solves 49 of 50 problems in TheoPhysEval physics benchmark
Overview
A team led by Suvrat Raju has published TheoPhysEval, a benchmark of 50 theoretical physics research problems, and reports that GPT-6-Astra, running in OpenAI's Codex, solved 49 of them, or 98 percent, with no human hints.
The authors say the cost was under $3 per problem and just under $116 for the full benchmark.
The authors describe the problems as resembling first research projects for PhD students. They also say the models may have memorized much of the theoretical physics literature, a possibility that could affect how the scores should be read. The problems and test setup are available as a dataset on Hugging Face for anyone who wants to check the results.
Written by AI from the articles below · updated Oct 15, 12:09 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- The DecoderNewsPickGPT-6-Astra solves 49 of 50 physics research problems at $2.40 apiece
AIA physics benchmark of 50 routine research problems tested commercially available AI agents without human hints. GPT-6-Astra, running in OpenAI's Codex, solved 49 of 50 problems, or 98 percent, at under $3 per problem and just under $116 for the whole benchmark. The authors say the problems resemble first research projects for PhD students, and that the models may have memorized much of the theoretical physics literature.
Heat trend
Not enough continuous observations to show a trend yet.