#Eval/Benchmark

AILiquid AI shared a Hugging Face guide to multi-harness reinforcement learning for coding agents, in which a proxy records the token ids and logprobs vLLM samples so training works without changing the harness. Per the quoted post, LFM2.5-2.6B rose from 42% to 54% after training across four harnesses at once, and imitation fine-tuning on 3,189 rollouts from Qwen3.8-27B plateaued at 47.5%, below both RL runs. The proxy, trainer, tasks, SFT data, training code and seven trained models are described as open.

AIHamel Husain recommends binary pass/fail labels over 1-5 Likert ratings when evaluating AI systems. He argues that binary labels force clearer thinking, produce more consistent labeling, and are significantly less complex to operationalize.

AIThomas Wolf shares Kevin Buzzard's reflections on whether mathematics is about human understanding and what exponential AI growth means for an infinite field. He quotes Buzzard's view that machines may eventually reach a natural boundary where further progress is not worth the resources, and that humans would then take over from there.

AILucas Beyer calls SWE-sweep a useful new benchmark, where agents must find and fix bugs in a repo checked out at an earlier commit, scored against unit tests from real later bugfixes. He notes two limitations: a model may find valid bugs that don't match the tested ones, and the construction makes training on the test set easy. He advises not overemphasizing small ranking differences once models score highly.

AICool eval. Simply ask an LLM “Land or Water?” and give it a latitude and longitude coordinate as text. Ask 16,200 times, plot as image. The models know. From compressing the internet.

AIServiceNow CoreAI introduced AutoSynthData, which uses a target model's failures and a stronger teacher's successes to generate and validate new agent training tasks. In EnterpriseOps Gym experiments, the Hybrid domain produced 2,000 samples and raised Gemma-4-26B-A4B-it mean Pass@1 by 7.2 percentage points, while the ITSM domain produced 1,994 samples and raised it from 18.77% to 27.18%.

Why it matters: The post shows how failure analysis, teacher demonstrations, and verifier checks combine into a repeatable pipeline for generating targeted agent training data.

AICoreWeave's new RL rollouts service uses ModelExpress and Router in NVIDIA Dynamo to speed up model weight reloads during RL post-training with minimal downtime. Working with NVIDIA and You.com, CoreWeave achieved 15× faster model reloads than its baseline while post-training Nemotron 3.5 Lightning. The speedup addresses GPUs sitting idle while inference workers wait to load updated weights between training iterations.

AIApple Machine Learning Research reports that discrete diffusion steps match the training distribution only when simultaneously written token positions are conditionally independent given already-fixed tokens. The authors show that per-position distributions cannot determine such dependence, and on the synthetic ScanAndAdd task, confidence-ranked groups of two or more positions were dependent and produced a generated distribution 29 times the sampling-noise floor in total variation.

AILiveKit now supports Microsoft's MAI-Transcribe-2-Streaming, MAI-Voice-2.1, and MAI-Voice-2.1-Flash for building voice agents. MAI-Transcribe-2-Streaming debuts at #1 on the Artificial Analysis accuracy leaderboard. Developers can pair it with MAI-Voice-2.1-Flash to build efficient and expressive voice agents.

AIGLM 5.3 and GLM 5.3 Flash are now available in Cursor. GLM 5.3 Max is the best-scoring open-weight model on CursorBench 4.0.

AIPerplexity open-sourced pplx-decider-27b, a state-of-the-art multimodal Decision Model, and offers it through a new Decisions API. Input tokens cost $0.04 per million and output tokens are free, with further price cuts promised in the coming days.

AIMike Knoop notes that a verification of the roughly 27B-parameter model is notable because it is about as large as fits within the official ARC Prize Kaggle competition runtime. ARC Prize reports Qwen3.8-27B from Alibaba's Qwen team scored 42.4% on ARC-AGI v2 at $0.45 per task and 87.5% on v1 at $0.22 per task.

AIARC Prize has published full ARC-AGI results for Alibaba's Qwen3.8-27B, with the testing run hosted by Baseten. The post links the public leaderboard, the open-source benchmarking repository for reproducing the results, and the testing policy.

AIJev, a reranker from TypeSafe AI, improved Wrangle's main-app search by 30.64% in matches per 200 candidates while costing 75% of the previous reranking cost. The gains were measured on searches recruiters actually run, rather than standard 10-candidate benchmarks, and the full benchmark is slated for release soon.

AIIn a business risk monitoring test by @edwardirby, the Jev judge matched the report quality of an ordinary LLM judge while missing no investigations, versus the LLM missing 5 of 11. The LLM's threat scores also flip-flopped from 0.35 to 0.68 to 0.50 on the same threat, while Jev was 250x cheaper and 3-6x faster.

AIFrançois Chollet argues the key difference between base LLMs and modern LRMs is a shift from transductive answer prediction to inductive prediction of the program or reasoning chain behind an answer. He says this enables test-time induction and substantial fluid intelligence in LRMs, which he claims base LLMs largely lack. He cites ARC 1 results: base LLMs remain around 10-15%, while LRMs of the same size or smaller saturated the benchmark in 2025.

AIFireworks AI's Ember-1 produced a near-identical composition to Kimi K3 on the same JavaScript painting prompt, according to Sophia Yang. Ember-1 used 9x fewer thinking tokens and ran 4x faster at half the cost.

AIGuillermo Rauch argues that the future is verification engineering, spanning proofs, end-to-end tests, benchmarks, and linters. He expects some of these tests to be deterministic and others agentic, and he says the approach looks great. The quoted post introduces e2e, an open-source agentic testing framework that mixes deterministic and agentic APIs and runs locally or in CI.

AIOdyssey introduces PROWL-2, a recursive loop that delivers up to 91% relative gains over the StarCraft world-model baseline. The post also reports stronger robot coordination in simulation.

AIGoodfire reports that its method catches more unsafe biological requests than frontier model safeguards while dramatically reducing false positives. The claim is illustrated by a plot in the first tweet of the thread.

AIGoodfire built monitors that let AI agents perform more biology work while flagging dual-use risks. The monitors reportedly outperform frontier model safeguards, with fewer refusals on dual-use tasks, and run fast and in real time while staying robust to adversarial attacks.

AIPrime Intellect reports that after about 100 GRPO steps on Hosted Training, Qwen3.6's reward on held-out problems rose from 0.127 to 0.361, a 2.8x gain. Qwen3.5, trained the same way, reached 0.356, suggesting the method works across model families. Both post-trained models finished well ahead of other open models and narrowed the gap to Claude Opus 4.8, with Qwen3.6 activating only 3B parameters per token.

AIMicrosoft AI launched MAI-Transcribe-2-Streaming, which Artificial Analysis ranks #1 of 38 models for final transcript accuracy at 2.5% WER, returned 0.13s after end of speech. Artificial Analysis lists its streaming price at $0.54 per hour of audio, at the higher end among leading streaming models. Microsoft's post claims the model is 55% faster and 60% cheaper than ElevenLabs and invites developers to build agents on its platform.

AILlamaIndex introduced Extract v2.5, a series of agents tuned for document extraction across cost-effective, agentic, and agentic plus tiers. The company reports the agents outperform Opus 5.5 and GPT-6 Sol while costing 30% to 4x less, with accuracy gains on long lists (86.1% to 95.5%), multi-page records (85.5% to 96.5%), and scanned forms (90.9% to 95.7%) on its agentic tier. The release adds advanced citations with bounding boxes and structural reasoning, and the agents are available on LlamaParse.

AIHugging Face shows that training LFM2.5-2.6B with RL inside the agent harnesses themselves lifted held-out task success from 42% to 54% across four harnesses. Before training, the model solved 62% of tasks in Mini-SWE-Agent but only 33% in Claude Code, so the same model behaved very differently per harness. The approach uses an OpenEnv capture proxy to record tokens and logprobs, Harbor for tasks and sandboxes, and TRL's async GRPO trainer, with 31% fewer tool calls on already-solved tasks; training in OpenCode alone mostly improved OpenCode.

AICloudflare released Clef and Clef-flash, two decision models hosted on Workers AI and open-sourced on Hugging Face under Apache 2.0, and launched a reinforcement learning fine-tuning service. In Cloudflare's tests, Clef classified a domain in 2.2s versus 4.7s for gpt-oss-120b, and the models are Jev-API compatible. The company is offering fine-tuning first through a forward-deployed engineering team, with a self-serve platform planned later.

AIHamel Husain advises against building automated evaluators for every failure mode discovered during AI development. He argues that teams should weigh the costs of different evaluator types, such as code-based checks versus LLM judges, before building an eval.

AIAlibaba's Wan 3.0 ranks #1 overall on the new Artificial Analysis AA-Video-T2V v2.0 text-to-video leaderboard, priced at $12 per minute of video. The benchmark judges models at 1080p using over 68,000 human preference votes across 1,000 prompts, and the author states Wan 3.0 leads 10 of 20 category boards.

AIGoogle announced Gemini 4 Argon, a model it says is built to sustain deep reasoning across complex, long-horizon workflows, roughly seven months after its last flagship release in February. The article says cybersecurity testing will be completed after October 1, with no benchmark scores, pricing, or availability details provided.

AILangChain built a model router inside its open source coding agent Open SWE that picks one of three models for each thread. In an A/B test against always using GPT-6 Astra, the median cost per thread fell 64% with no measurable change in merged PR rate. The router runs on the thread's first message, using a base prompt, per-tier criteria, and a classifier model, and the post lists next steps including subagent routing and mid-thread re-routing.

AIHamel Husain has published a longer review of a new Claude Auto Eval plugin after many users asked about it. He invites readers to share their experiences using the plugin and how it went for them. The plugin is part of Claude's ability to help build evaluations and hillclimb on them, as described by @ClaudeDevs.

AIGoogle has announced Gemini 4 Argon, initially available only to trusted cyber defenders through its Fairwind Program while US government approval is pending. The author says the model is aimed at long-running software engineering, enterprise knowledge work, and cybersecurity tasks, with a 1 million token output limit. The post also gives promotional pricing of $2 per million input tokens and $10 per million output tokens, rising to $4 and $20 afterward, alongside a benchmark comparison.

Why it matters: The post places Gemini 4 Argon's benchmark table beside GPT-6 Astra and Claude models, showing where each leads across coding, knowledge work, and cybersecurity tasks.

AIVarun Mohan says Gemini 4 Argon is doing well on the AA Coding Agent Index when run with the Antigravity harness. He notes that real-world usage matters more than benchmarks and promises more to come.

AIUnder equal time budgets and the same frontier LLM backbone, a single session of a minimal-harness coding agent with read, write, and bash primitives matched open-source state-of-the-art autonomous machine learning engineering harnesses. Apple researchers found the added orchestration and retrieval machinery redundant in large-scale ablation studies, pointing to the backbone model as the main driver of performance. They conclude that hand-crafted harnesses around strong models yield poor returns on current MLE benchmarks.

AIThe CDC announced that Google's science AI model ranked highest among 39 eligible models for forecasting flu-related hospital admissions during the 2025-26 flu season. Google's forecasts were built with Empirical Research Assistance, an AI tool that generates computational solutions across scientific fields.

AIGoogle DeepMind introduced Gemini 4 Argon, a new frontier model built for coding, enterprise knowledge work, and cybersecurity defense, rolling out to trusted testers through its Fairwind Program. Yi Tay says Gemini 4 outperforms astra and fable on many tasks, though the post gives no benchmark figures.

AIDeedy argues that benchmark scores for frontier AI models are increasingly meaningless because labs tune for them before launch. He suggests trusting price instead: a high price indicates a genuinely good model, while a low one suggests the model is benchmaxxed.

AIThe post quotes Google's Sundar Pichai introducing Gemini 4 Argon as an early look at the next model. It claims frontier performance in complex workflows, cyber defense, and software engineering, and says Google teams are using it for tasks from coding to quantum computing. The author adds that on FrontierSWE the model is very self-critical and often says "Eureka!", a personality they describe as a large improvement over previous Gemini models.

AIYuchen Jin suggests Google is back in the AI race after a model reportedly outperforms Astra and Opus 5.5 across the board on benchmarks. He adds a caveat that the results may reflect benchmark optimization rather than real capability gains, and says he would welcome Google rejoining the race.