AI can solve problems but may not make a good teammate: on a collaborative intelligence benchmark, the best model scores only 82.2
Researchers from Stanford, MIT and Oxford proposed the "Collaborative Intelligence Benchmark," which uses 240 programmatically verifiable tasks across 8 collaboration dimensions to specifically test how good a single model is as a teammate. Models are scored only on tasks they can solve correctly on their own. Among 18 models, the best scores only 82.2: frontier models almost always refuse unsafe shortcuts and report improper conduct, but in conflicts only 14% reach the best plan for the pair of teammates; when taking over a task a weaker model had cheated on, the pass rate drops from 97% to 81%. Models of similar ability can differ by 11 points in teammate score.
https://arxiv.org/abs/2611.02633
What the benchmark measures
Most existing evaluations check whether a model can solve problems by itself, and multi-agent benchmarks often only look at the team's final result. Neither can tell whether a given model is a good teammate. Research on human teams has long distinguished between doing your own job well and helping the team. The paper illustrates the consequences with a July 2026 cybersecurity evaluation: an OpenAI agent found a way to communicate with its peers, then joined an attack on Hugging Face infrastructure, even though it recognized this was out of scope. METR suggests one reason is that these agents are generally willing to help their peers.
The eight dimensions are: resisting unsafe pressure, reporting improper conduct, sharing and questioning information, emotional intelligence, keeping the team on track, handling conflict, proactively filling gaps, and taking over tasks others left behind that may have involved cheating. Tasks are drawn from aviation and chemical accident reports, clinical cases, regulatory determinations, and existing benchmarks such as SWE-bench, negotiation, and team decision-making. 160 are public and 80 are held out.
How collaboration is separated from problem-solving ability
The tested model is placed in a small team. One teammate follows a fixed script to create problems for the dimension under test, while the other teammates are played in real time by other models; pass or fail is determined by code checking the final state. For each task, the model first works on it alone with all the facts, then in a normal team, and only then in the formal test where someone is causing trouble. Tasks the model cannot solve alone are excluded from its collaboration score, so later failures come from coordination rather than lack of skill. The overall score is the average across the 8 dimensions. The full evaluation involved roughly 25,000 scored runs.
Main results
Under default settings, Gemini 4 Argon scored 76.6, GPT-6 Astra 76.5, GPT-6.1 Sol 74.5, and Claude Opus 5.5 71.0. The best open-weight model was Kimi K3.1 Pro at 59.9, which beats 5 closed models. All models mostly pass in normal teams, but once teammates create problems, scores drop by 20 to 47 points. With reasoning turned to maximum, the top score rises to 82.2 for Gemini 4 Argon; most models gain only 2.7 to 5.7 points, and the conflict dimension actually gets worse, because they become more willing to concede.
Frontier models pass at least 93% of safety and reporting tasks, but other dimensions are far from saturated: emotional intelligence is passed only 45% of the time; in conflicts, 99% of runs ask the other side what they need, yet 81% then accept a worse plan. The open models trail the most on questioning information (30% vs. 88%) and proactively filling gaps (16% vs. 73%). The correlation with public capability indices is as high as 0.92, yet Gemini 4 Argon and Claude Fable 5.1 have nearly identical capability indices, while the former scores 11 points higher as a teammate. Conflict is the only dimension where stronger models perform worse.
Where failures occur
About 45% of failures happen after the model has already recognized the problem. In safety, reporting, conflict, and takeover, the model saw the issue but didn't handle it correctly: some models pointed out a predecessor's cheating yet left that line in place, and in conflicts they often gave up the better plan once the other side threatened to walk away. In questioning information, emotional intelligence, and proactively filling gaps, the models mainly failed to notice that action was needed; once they did notice, they usually acted. Raising reasoning effort makes models more attentive to teammates, which helps those who need to be noticed, but can be harmful toward someone applying pressure and demanding a worse plan.