# Ai2 finds frontier LLMs judge research idea novelty inconsistently

> Original title: LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it?

- Source: Ai2
- Published: 2026-10-27T16:36:50.000Z
- Bad Influence: https://badinfluence.news/items/wbyu414rq3vaj6nk8r8teumkx
- Original: https://x.com/allen_ai/status/2115120549680521403

## Summary

Ai2 released a new eval showing that frontier LLMs can reach different novelty conclusions about similar AI-generated research ideas depending on how they are asked to evaluate them. The eval is published at github.com/allenai/novelty-eval.

## Post

LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it?

Our new eval reveals even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them. 🧵

https://github.com/allenai/novelty-eval

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHVpq7mSX0AEbIWi.jpg&mode=thumb&exp=1793404800&sig=465845a0599cb4fa)