Skip to content
View original post on X: Rohan Paul
Published
63/100

LLMs catch more planted errors in clinical trial papers than untrained human reviewers

AIA preprint tested 15 large language models on planted methodological errors in clinical trial reports, with untrained human reviewers averaging 27% major-error detection. The models scored 53% to 96% on the original reports and 58% to 99% on 58 trials published in 2026, with Claude Opus 5.5 highest at 99%.
Rohan PaulVerified on X
@rohanpaul_ai
View on X

AI now catches far more errors in clinical trial papers than human peer reviewers: all 15 models tested scored 53% to 96% on planted errors, while untrained human reviewers averaged just 27%.

The result held on 58 new trials published in 2026, so the models weren’t simply recalling old papers they had seen in training; Claude Opus 5.5 scored highest at 99%.

Humans miss the errors that most change a trial’s meaning, catching text-table contradictions only 7% of the time, and they take 5 to 8.5 hours per review against AI’s 1.4 to 16.3 minutes.

Ruslan Rust@rust_ruslan
Lots of debate this week on whether journals should ban AI in peer review. We actually tested this in our new preprint, and the results were pretty clear to me. For context, I've reviewed for most of the big journals and evaluated 1,000 human reviews as an editor. I love to read and evaluate science. TLDR: From now on I won't do any peer review without AI. For clinical trials, I'd even call human-only review dangerous 🧵
View quoted post on X
Source details

Rohan Paul · x.comPublished

Original title: > AI now catches far more errors in clinical trial papers than human peer reviewers: all 15 models tested scored 53% to 96% on planted er...

Topics

Anthropic / ClaudePapers & researchEvals & benchmarksTrends