# Study finds research agents' experiment rankings add little beyond random picks

> Original title: Regarding autoresearch agents, this question has attracted a lot of attention.

- Source: Elvis Saravia
- Published: 2026-10-30T16:24:01.000Z
- Bad Influence: https://badinfluence.news/items/fbu527xyep1m9v1w6rvvfvdpo
- Original: https://x.com/omarsar0/status/2116204488562528713

## Summary

A study gave six frontier models seven recent NLP papers and ran all 415 experiments they proposed, including the ones ranked last. Of these, 194 were judged sound, valuable, and new, and only 73 were also beyond the seed paper and non-obvious. First-choice rankings beat a random pick but not the first experiment written, and forecasts beat a base rate without clearly outperforming a fresh model instance.

## Post

Regarding autoresearch agents, this question has attracted a lot of attention.

Do research agents have taste?

This has attracted massive interest because autoresearch agents could unlock major discoveries.

This study gave six frontier models seven recent NLP papers, then ran every experiment they proposed, including the ones they ranked last.

Of 415 experiments, 194 were judged sound, valuable, and new. Only 73 were also beyond the seed paper and non-obvious.

The rankings added little. First choices beat a random pick, but they were not shown to beat the experiment written first.

Forecasts beat a base rate, but they were not measurably better than a fresh instance of the same model that read only the paper and registrations.

They tested with teams of agents, which cost more, with no clear improvement in proposal quality.

https://arxiv.org/abs/2610.40488

Chat with Paper: https://academy.dair.ai/papers/rtaste-do-research-agents-have-taste-evaluating-early-stage-research-judgment-2610.40488

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHV5ExDlaAAIMErD.jpg&mode=thumb&exp=1793577600&sig=8e24aa0bbdd374d5)