Topsy-turvy test-time compute
Closed models refuse more at max reasoning effort, lowering accuracy
AISummary
Lewis Tunstall (@_lewtun) reports that when evaluating closed models on refusal-triggering tasks such as cyber and bio, low reasoning effort yields fewer refusals and higher accuracy than max effort. He notes this produces test-time compute curves where max effort performs worse than low, and asks how to fairly report such results.
Post on XView on X
Lewis TunstallVerified on X
@_lewtun
Here's a quirk when evaluating closed models on tasks that trigger refusals (cyber, bio etc): > low reasoning effort => few refusals, high accuracy > max reasoning effort => many refusals, low accuracy This gives you test-time compute curves where max effort does way worse than low because the refusals tank the performance. What is the fairest way to report performance in these cases?View quoted post on X
Source: Lewis Tunstall · x.comPublished · added here