Ai2 finds LLM novelty judges are unreliable on AI-generated research ideas
AIAi2 releases a paper, benchmark, data and code testing how reliably six LLMs judge whether research ideas are novel. Changing a few prompt words shifted one model's accuracy on identifying the more novel of two AI-generated ideas by more than 50 percentage points.
Loading source posts…
This story is featured in top picks: “Ai2 finds LLM novelty judges give unstable verdicts on research ideas”
Source details
Hugging Face Blog · huggingface.coPublished · added here
Original title: Can we trust LLMs to judge whether a research idea is novel?