Study finds six frontier models' experiment rankings add little beyond random picks

A study gave six frontier models seven recent NLP papers and ran all 415 experiments they proposed, including those they ranked last, according to Elvis Saravia, who describes the arXiv paper 2610.40488.

Of the 415 experiments, 194 were judged sound, valuable, and new, and only 73 were also beyond the seed paper and non-obvious.

The models' first-choice rankings beat a random pick, but the study did not show they beat the experiment written first. Forecasts beat a base rate without clearly outperforming a fresh model instance, Saravia says.