# Prime Intellect details flaws in public AI benchmarks and their fixes

> Original title: Learnings from Auditing Public Evaluations

- Source: Prime Intellect Blog
- Published: 2026-11-03T05:00:00.000Z
- Bad Influence: https://badinfluence.news/items/wuqmik2x3627i34hgpwbqkztv
- Original: https://www.primeintellect.ai/blog/auditing-public-evaluations

## Summary

Prime Intellect says it found grading, tool-access, and reward-hacking problems while running popular evaluations reported in model cards. It applied fixes to AutomationBench, OfficeQA Pro v2, and HLE-Diamond, and open-sourced them. For AutomationBench, the post reports Claude Opus 5.5 at 92.15% partial and 70.83% strict under Zapier's mode, versus 91.61% and 69.50% for DeepSeek V4.1-Flash 1017; it says the scores are not directly comparable across modes.

## Why it matters

The post traces how specific grading, tool access, and monitoring flaws distort benchmark scores, showing what to check before trusting published evaluation results.