Skip to content
View original post on X: Prime IntellectOfficial
Published
22/100

OfficeQA Pro v2 replaces brittle regex grading with a judge

AIPrime Intellect says it replaced OfficeQA Pro v2's regex-based answer grading with a judge after the original script proved brittle. It also says the benchmark's web access requirement let some models cheat, and that the fixes changed the model rankings.
Prime IntellectVerified on XOfficial
@PrimeIntellect
View on X

Part of a thread · earlier post

OfficeQA Pro v2 is a search benchmark using financial documents.

Their grading script was rather brittle, relying on regex to detect the answer. We replaced it with a judge.

Furthermore, it required web access, which some models use to cheat. After fixing, ranking changed.

Source details

Prime Intellect · x.comPublished

Original title: OfficeQA Pro v2 is a search benchmark using financial documents.

Topics

Data & trainingEvals & benchmarks

Tags

#Search