Max Planck study finds reasoning increases shared errors across LLMs

Max Planck researchers report that turning on reasoning makes nine Qwen and GPT models agree more across three US census prediction tasks.

The share of people that all nine models got wrong rose by 2.0 to 2.4 percentage points. For cases with the longest reasoning traces, adjusted agreement fell to about 0.31 to 0.39, according to a post by Rohan Paul summarizing the study.

The only source so far is that summary post, which reports the paper's findings. The post's own evidence line describes long traces as marking where the models disagree, so the agreement figures for those cases may be read as the point where the models split.