New Tsinghua, Oxford, and Stanford paper finds that LLMs keep reasoning out loud even with thinking turned off, especially on open-ended questions.
With thinking disabled, DeepSeek-V4-Flash still wrote out reasoning in 99.9% of open-ended answers. A missing think tag or a short reply does not prove the model skipped reasoning.
Yes/no questions are easy to answer directly, multiple-choice questions sit in the middle, and open-ended ones keep pulling the model back into reasoning.
Forcing answer-only replies on open-ended tasks raised compliance to about 40% across 5 models but cut accuracy by about 15 points.
– arxiv. org/abs/2610.11765
Title: "Thinking Inertia: LLMs Keep Thinking When Told Not To"