Decision models are poorly calibrated on knowledge and reasoning tasks. Across almost all probability levels for chosen answers, accuracy falls below that probability for all decision models. This gap is largest on AA-Omniscience-Decision and HLE-MultiChoice, while most models are relatively well-calibrated on WildGuardTest (classification) and When2Call (tool calling)
Decision models are poorly calibrated on knowledge and reasoning tasks
AISummary
Artificial Analysis reports that decision models' accuracy falls below their stated confidence for almost all probability levels on knowledge and reasoning tasks. The gap is largest on AA-Omniscience-Decision and HLE-MultiChoice. Most models are relatively well calibrated on WildGuardTest and When2Call.
Post on XView on X
Source: Artificial Analysis · x.comPublished · added here