# Ai2 finds LLM novelty judges give unstable verdicts on research ideas

> Original title: Can we trust LLMs to judge whether a research idea is novel?

- Source: Ai2 (Allen Institute for AI)
- Published: 2026-10-27T08:00:00.000Z
- Bad Influence: https://badinfluence.news/items/falc0dlq0530ocj5nkpjbqztm
- Original: https://allenai.org/blog/novelty-bench

## Summary

Ai2 releases Old Ideas, Novel Problems, a study of how reliably LLMs judge the novelty of research ideas, along with a benchmark, data, and code. Across six LLM judges, changing a few prompt words shifted accuracy on AI-generated ideas by more than 50 percentage points. A simple prompted Claude Opus 4.6 baseline outperformed two purpose-built novelty evaluators while costing far less.

## Why it matters

The study shows that small prompt and setup changes can flip LLM novelty judgments, so readers evaluating AI-generated research ideas should check the judge setup.