# Ai2 finds LLM novelty judges are unreliable on AI-generated research ideas

> Original title: Can we trust LLMs to judge whether a research idea is novel?

- Source: Hugging Face Blog
- Published: 2026-10-27T16:12:42.000Z
- Bad Influence: https://badinfluence.news/items/wslpqgcia3nn092ico4ijfo7w
- Original: https://huggingface.co/blog/allenai/novelty-bench

## Summary

Ai2 releases a paper, benchmark, data and code testing how reliably six LLMs judge whether research ideas are novel. Changing a few prompt words shifted one model's accuracy on identifying the more novel of two AI-generated ideas by more than 50 percentage points. A simple prompted Claude Opus 4.6 baseline outperformed two purpose-built novelty evaluators while costing far less.