Ai2 study finds LLM novelty judges give unstable verdicts on research ideas

1. 12:36 PM
   Ai2@allen_aiOfficial

   Ai2 finds frontier LLMs judge research idea novelty inconsistently

   AIAi2 released a new eval showing that frontier LLMs can reach different novelty conclusions about similar AI-generated research ideas depending on how they are asked to evaluate them. The eval is published at

   
2. 12:12 PM
   Hugging Face BlogOfficialPick

   Ai2 finds LLM novelty judges are unreliable on AI-generated research ideas

   AIAi2 releases a paper, benchmark, data and code testing how reliably six LLMs judge whether research ideas are novel. Changing a few prompt words shifted one model's accuracy on identifying the more novel of two AI-generated ideas by more than 50 percentage points. A simple prompted Claude Opus 4.6 baseline outperformed two purpose-built novelty evaluators while costing far less.

3. 4:00 AM
   Ai2 (Allen Institute for AI)OfficialPick

   Ai2 finds LLM novelty judges give unstable verdicts on research ideas

   AIAi2 releases Old Ideas, Novel Problems, a study of how reliably LLMs judge the novelty of research ideas, along with a benchmark, data, and code. Across six LLM judges, changing a few prompt words shifted accuracy on AI-generated ideas by more than 50 percentage points. A simple prompted Claude Opus 4.6 baseline outperformed two purpose-built novelty evaluators while costing far less.

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.