# OfficeQA Pro v2 replaces brittle regex grading with a judge

> Original title: OfficeQA Pro v2 is a search benchmark using financial documents.

- Source: Prime Intellect
- Published: 2026-11-03T18:54:48.000Z
- Bad Influence: https://badinfluence.news/items/ds2qg0brl605lwb0pj6ha3x1h
- Original: https://x.com/PrimeIntellect/status/2117691982202155058

## Summary

Prime Intellect says it replaced OfficeQA Pro v2's regex-based answer grading with a judge after the original script proved brittle. It also says the benchmark's web access requirement let some models cheat, and that the fixes changed the model rankings.

## Post

OfficeQA Pro v2 is a search benchmark using financial documents.

Their grading script was rather brittle, relying on regex to detect the answer. We replaced it with a judge.

Furthermore, it required web access, which some models use to cheat. After fixing, ranking changed.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWOISppbMAE6LKF.jpg&mode=thumb&exp=1793923200&sig=46189b4492c1ab84)