Trending storyDeveloping
Closed models refuse more at max reasoning effort, lowering evaluation accuracy
2 articles1 sourcesince Oct 14Last article 1h ago ·
Article timeline
The articles in this story. Times are ET.
Oct 14
Lewis Tunstall@_lewtunXClosed models refuse more at max reasoning effort, lowering accuracyAILewis Tunstall (@_lewtun) reports that when evaluating closed models on refusal-triggering tasks such as cyber and bio, low reasoning effort yields fewer refusals and higher accuracy than max effort. He notes this produces test-time compute curves where max effort performs worse than low, and asks how to fairly report such results.
- Lewis Tunstall@_lewtunXClosed models refuse more at max reasoning effort, skewing evaluations
AILewis Tunstall says closed models on refusal-triggering tasks such as cyber and bio show few refusals and high accuracy at low reasoning effort, but many refusals and low accuracy at max effort. He notes this makes max effort score worse than low effort, and asks what the fairest way to report such performance is.