#Eval/Benchmark

AIOpenRouter has released Microsoft-Decision-1, a model Microsoft says posts the highest accuracy across 36 blind benchmarks of about 150K questions. Microsoft says it runs 4.5x faster than the runner-up and 35x faster than GPT-6 Sol, with decisions flipping on only 1.3% of perturbed inputs. The model is post-trained from Qwen3.5-9B, costs $0.042 per million input tokens, has free output and a 32K context window.

AIStepFun's Step 5 Preview, a 600B-total, 27B-active MoE model with 1M context and vision, is free to try on Nous Portal for one week. Nous Research says it scored 33.89 on the Hermes Index, the same score as GPT-6 Luna.

AIMistral releases Voxtral Mini 4B Realtime Arabic, a streaming speech-to-text model for Arabic dialects and Modern Standard Arabic under the Apache 2.0 License. The model has about 4.4 billion parameters, is fine-tuned from Voxtral-Mini-4B-Realtime-2602, and reaches an average 8.82% Character Error Rate across seven Arabic benchmarks at a 480 ms transcription delay. It can be run with vLLM or Transformers 5.2.0 or later.

AIMistral AI has released LIDstral-Arabic, a fast classifier that identifies Modern Standard Arabic, Arabic dialects, and non-Arabic languages written in Arabic script across 51 classes. On Moroccan Darija, it scores 88.67% F1, versus 72.65% for LahjatBERT ALDi CL and 71.28% for GlotLID v3, across 84,870 evaluation examples. The model runs on CPU and is available from a private Hugging Face repository under Apache 2.0.

AIJetBrains released Mellum2.1, a 12B mixture-of-experts coding model with 2.5B active parameters under Apache 2.0, emphasizing agentic programming. Under high load, its inference throughput in tokens is nearly twice that of Qwen3.5-9B in JetBrains' comparison, and multi-token prediction (MTP) speeds single-request responses by about 1.6x. The model is available on Hugging Face for local or private-infrastructure deployment, with GGUF and vLLM MTP support announced for later.

AIHarrison Chase says trajectory labeling should be split into several questions rather than one pass/fail judgment. LangChain's Jev judge in LangSmith Evals answers difficulty and correctness as separate typed outputs in a single pass on every trace.

AIAnthropic released Claude Haiku 5.5, which the article says outperforms DeepSeek V4.1 Flash and GLM-5.3-Flash on official benchmarks and matches GPT-6 Luna on price. On OSWorld 2.1, its Low effort tier scores 42.0% at $0.07 per task, versus 15.7% at $1.45 for Haiku 4.5 at Max. Migrating from Haiku 4.5 requires changes to thinking configuration, sampling parameters, assistant prefill, and the computer-use tool version.

AIAnthropic released Claude Haiku 5.5, a small model priced at about a quarter of Haiku 4.5's running cost. Anthropic also halved Sonnet 5.5 cache read prices from 20 cents to 10 cents per million tokens.

AIXiaomi MiMo has released MiMo-V2.5-ASR, an open-source speech recognition model that the company says achieves state-of-the-art results across multiple benchmarks. The model supports bilingual Chinese–English recognition, Chinese dialects such as Wu, Cantonese, Hokkien, and Sichuanese, code-switching, and lyrics transcription. It is also designed to handle noisy environments and multi-speaker conversations.

AIMiniMax says its open-source H3 model is almost on par with closed-source state-of-the-art video world models on physics. The claim is supported by a quoted benchmark, World Models' Last Exam in Physics, where eight leading models scored at most 57.76/100 across 40 physics tasks, and free-fall videos averaged only 26.61/100 on composite scores.

AIGoogle Research says results from evaluating its research medical system AMIE in a real clinic, with BIDMC, are published in The Lancet. Across 100 patient interactions, AMIE recorded 0 safety stops and matched doctor diagnoses in 90% of cases.

AIArena has released its Alignment Index, a ranking that tracks AI models for unauthorized actions, deceptive completions, and false attribution of intent. The full technical report and rankings are available on Arena's blog and leaderboard site.

AISherwin Wu, an OpenAI employee, says the updated Harvey LAB-AA v1.1 benchmark, announced by Artificial Analysis with Harvey, is more useful than the original LAB results. The new Hallucination-Gated All-Pass Rate credits a task only when every rubric criterion passes and no material hallucination appears. Grok 4.7 (xhigh) leads at 9.4%, while GPT-6 Astra (max) at 8.6% has very few material hallucinations.

Why it matters: The update adds a hallucination gate to a legal benchmark, showing that models with high all-pass rates can rank much lower once material errors count.

AIThe Association of Historical Mathematicians (AHM) has called for a boycott of OpenAI after the company released more than 700 AI-generated proof files at once. Fields Medalist Terence Tao, who chairs the group, argues that AI solving open problems autonomously reduces seminars, collaborations, and fertile research directions, and that the field should shift its measure of progress toward explanation and community-building.

Why it matters: The article links the AHM boycott call to Tao's argument that AI-driven proof volume is changing how mathematicians measure progress and whether solutions remain useful.

AIPerplexity's Decider V1.1 ranked first on DecisionBench while also having the lowest cost, according to a post highlighting the result. The benchmark results cited include 949 shared text cases, 93.9% accuracy, a 534 ms median latency, and $0.016 per 1k decisions.

AIJetBrains has released Mellum2.1, a 12B mixture-of-experts thinking model with 2.5B active parameters, under Apache 2.0 on Hugging Face. Post-training reinforcement learning in real software repositories raised SWE-bench Verified from 2.0 to 47.0, according to JetBrains' self-reported results. Qwen3.5-9B still leads on SWE-bench Pro, GPQA Diamond and AIME, and GGUF builds start at 7.0 GB for local use.

AIAnthropic released Claude Haiku 5.5, raising its OSWorld 2.1 score from 15.7% to 72.4% and supporting a 1 million token context window. The article notes Haiku 5.5 still scores 39.2% on Terminal-Bench 4.0 against Sonnet 5.5's 70.6%, and that prompts above 100,000 tokens are priced higher, so migration costs need to be measured on real workloads.

AIPerplexity's pplx-decider v1.1 scored 643 of 669 on clinical decisions versus 628 for Jev, according to a quoted post. The quoted post says it costs 42% less per decision at about the same speed and that its weights can be downloaded and run inside hospitals.

AIStepFun has released Step 5 Preview, its flagship model for agentic work, built on a sparse Mixture-of-Experts architecture with 27B active and 600B total parameters. The source says it performs strongly in software engineering and professional tasks, but the feed supplied only an excerpt, so benchmark details are not available here.

AIJetBrains released Mellum2.1, a 12B mixture-of-experts model with 2.5B active parameters under the Apache 2.0 license, built for coding agents. Post-training shifted to reinforcement learning across thousands of environments and millions of sandboxed runs, and the model is available on Hugging Face. The source reports gains over Mellum2 on LiveCodeBench, AIME, GPQA Diamond, BFCL v4, IFEval, and SWE-bench Verified, and says it serves almost twice the tokens of Qwen3.5-9B under heavy load.

Why it matters: The post shows how reinforcement learning in real sandboxed environments changed a compact open model's repository work, with benchmark gains against Mellum2 and two peers.

AIGoogle Developers Blog introduces AQuA, an ambient quality agent that runs in a customer's Google Cloud project and samples production sessions to find recurring agent failures. In a 32-session travel-concierge sweep, it verified six issues and traced two of them to specific prompt lines, and a replay after the fixes raised full-session passes from 5/32 to 13/32. The post notes that verification and diagnosis are model-based, and that the tool proposes edits without applying them.

Why it matters: The post walks through a concrete production workflow, from sweep and verification to a code-anchored fix and replay, that shows how to diagnose silent agent failures.

AIGPT-6 with Intelligent UI begins rolling out globally in the ChatGPT Chat tab for Plus, Pro, Business, and Enterprise users today. The rollout expands to Free and Go tiers starting tomorrow. Plus, Pro, Business, and Enterprise get GPT-6 Sol, while Free and Go get GPT-6 Luna.

AINVIDIA reports that fine-tuned Nemotron models reached gold-medal level at both IOI 2026, scoring 535.4 out of 600, and IMO 2026, scoring 30 out of 42. The IOI run was a live, unofficial, unsupervised benchmark, while IMO proofs were graded by official IMO graders. The post also releases checkpoints, datasets, a new 200-problem benchmark, and inference pipelines on Hugging Face and NeMo-Skills.

Why it matters: The post traces how SFT, RL, and a generate-verify-refine loop turned Nemotron into gold-level specialists for IOI and IMO, with the training and inference details shared.

AIAnthropic released Claude Haiku 5.5, which it calls its cheapest, fastest, and most capable small model. It costs around 75% less to run than Haiku 4.5 and is aimed at high-volume, cost-sensitive tasks such as summaries and classification. The release also cuts Sonnet 5.5 cache read prices by 50%, and the model is available on AWS, Google Cloud, and Microsoft Azure.

AIAnthropic released Claude Haiku 5.5, which scores 43 on the Artificial Analysis Intelligence Index, up 26 points from the last Haiku release. Pricing is $0.10/$0.50 per 1M input/output tokens up to 100k tokens, rising to $0.50/$2.50 above that, but at max effort it uses about 162k output tokens per Intelligence Index task, roughly 3x GPT-6 Luna.

Why it matters: The benchmark shows Haiku 5.5 scores well but uses far more output tokens than GPT-6 Luna, so cost per task matters beyond list price.