ServiceNow and McGill paper finds AI judges miss subtle mistakes in long computer-use tasks

ServiceNow and McGill researchers report that AI judges still miss subtle errors when reviewing long computer-use agent runs.

They built AgentHorizon, a benchmark of 1,373 judging tasks drawn from human screen recordings, and measured judges with balanced accuracy, where always answering "yes" scores 50%.

The best judge, GPT-5.5 in Codex, reached 80.9% balanced accuracy on the hard split. Qwen 3.6 27B scored 77.6% when reading the whole run directly but 70.4% when working as an agent with tools, according to the report, which says giving judges tools does not reliably help.