OpenAI finds agent safety evals and production behavior diverge, driven by objective awareness
AIOpenAI's alignment team compared agent success on Respecting Warnings evaluations with matched Codex production decisions across six models. Evaluation success was lower than production success for every model, with gaps of 10.2 to 37.4 percentage points, and rank order stayed positively correlated (Spearman's ρ = 0.771).
AIWhy it matters.
The study separates general eval awareness from awareness of the specific objective being tested, a distinction that changes how safety scores should be read.
Source details
OpenAI Alignment Research Blog · Joy Hsu, Sam Arnesen · alignment.openai.comPublished · added here
Original title: Eval-production behavior gaps in agent safety and the role of eval-awareness