Ai2 finds frontier LLMs judge research idea novelty inconsistently

LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it?

Our new eval reveals even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them. 🧵

https://github.com/allenai/novelty-eval