#Eval/Benchmark

AISenseTime released its SenseNova U1.5 technical report, describing an open-source 8B native MoT unified model that connects understanding and generation through shared attention. The model reports 68.2% on VBVR-Pro-Bench, ahead of Nano-Banana-Pro (56.4%) and GPT-Image-2 (50.7%), and its full training recipes, including SFT, RL, and multi-expert on-policy distillation, are open-sourced.

AINortheastern University MS student Soham Padia used Ai2's open Olmo 3-32B model to build Steering Arena, a public game in which players submit text prefixes to steer prosocial behavior. About 600 submissions from a few dozen people showed the top 36 entries were unreadable token strings, while the best plain-English entry ranked 37th at about 2.7 times lower score. The results suggest that once an evaluation metric is exposed, it becomes an optimization target.

AIDatabricks rolled out Astra to all of its engineers and reported internal measurements. Engineers given Astra increased overall coding spend by around 60% compared with baseline, and Astra outperformed prior top models on highly complex system design tasks, while the gain on medium or low complexity tasks was unclear.

AIMatei Zaharia says agent harnesses make a large difference in cost, even on open-source coding benchmarks, and Melissa Pan's research examines why. Her quoted evaluation of seven models across Claude Code, Codex, and Pi found harness choice had little effect on task success but significantly affected cost. A simple harness can be competitive, and the native harness is not always the best.

AITencent's WeVisDoc-4B, fine-tuned from Qwen3-VL-4B-Instruct, converts page images into structured Markdown with LaTeX formulas and HTML tables. It scores 95.38 Overall on OmniDocBench v1.6 and a mean Overall of 75.54 across three PureDocBench tracks, ranking first among compared end-to-end parsers in all four reported settings. The model is available on Hugging Face and runs through vLLM, which requires version 0.11.1 or later.

AITencent's WeVisDoc-4B, fine-tuned from Qwen3-VL-4B-Instruct, scores 95.38 Overall on OmniDocBench v1.6 and 75.54 mean Overall across three PureDocBench tracks. The end-to-end parser converts page images into structured Markdown with LaTeX formulas and HTML tables, and the 2B variant is also available. The repository provides vLLM serving scripts with a 32768-token default context and a Python client for batch processing.

AITogether AI says teams can migrate from closed to open-source models in weeks to months, not years, using a managed service. The guide recommends defining the use case, shortlisting candidates with relevant benchmarks, and replaying existing traffic against them to test accuracy, cost, and latency against current targets.

AIHiDream.ai launched vivago R1, a content creation agent, globally, with a domestic version upgrade. The company says R1 can output five-minute high-quality videos through agent planning, with a claimed 85% usable-output rate and support for multi-round extensions. It also released HiDream-O1-Video-1.0, a native omni-modal video model supporting single shots of 5 to 20 seconds at 1080p.

AIJeff Dean congratulated Periodic Labs on results from its materials-science AI work. The post itself provides no specific figures, so the announcement is limited to the congratulations.

AIPeriodic Labs says it used 1,300 H200 GPUs and months of its lab data to mid-train and RL an open-source model called Neon, which it claims surpasses GPT-6 Astra on its analysis benchmark. The company says it is focusing first on hard materials science problems, including superconductors, magnets, and semiconductor materials.

AIJason Wei argues that specialized, often private wet-lab data can let a task-specific model outperform a general frontier model on scientific tasks. He cites Neon, an open-source model that Liam Fedus says was mid-trained and RL-tuned on experimental data using 1,300 H200s to surpass GPT-6 Astra on an analysis benchmark. The post frames this data as a potential moat as work moves toward the frontier of science.

AIGoogle introduces Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking, two live dialogue models for voice agents that reason and speak simultaneously. The Extended Thinking version scores 82.6 on Artificial Analysis' Speech to Speech Quality Index and 97.7% on Big Bench Audio, while 3.8 Live targets scale and cost efficiency. Developers can access both through the Gemini API in Google AI Studio, and enterprise and consumer rollouts vary by product.

Why it matters: The source names the two models, their access paths, and specific benchmark results, showing how the voice agent capabilities differ between the two tiers.

AIPeriodic Labs chose SGLang and Miles to build Neon, an open-source model it says surpasses GPT-6 Astra on its analysis benchmark after mid-training and RL on 1,300 H200s. RadixArk says Periodic extended both frameworks for scientific RL at trillion-parameter scale, delivering more efficient training, lower memory use, and 2.5x faster inference. The work has been contributed back to both projects.

AIIn a Paint recreation test, GPT-5.6 Astra built the image from layered geometric shapes, while Qwen3.8 Max worked pixel by pixel. Qwen's output looks closer to the original, but Raschka argues this single example does not show either model generalizes better or has stronger computer-use or visual understanding, and it illustrates how benchmarks comparing only final results can be misleading.

AITencent Hunyuan introduced EvolveScaler, a benchmark that builds worlds as executable state machines and renders them into natural language with 117 prototypes, 159 question operators, and five difficulty tiers. On the hardest tier, 14 frontier models' median avg@5 falls to 11.3. Training on EvolveScaler data yields a +5.25 average gain across 8 out-of-distribution benchmarks.

AIShanghai AI Laboratory's Intern Large Models announced Intern-S2-397B, available in BF16 and FP8 under Apache 2.0. The post reports 87.0 on FrontierScience-Olympiad and 84.0 on SWE-bench Multilingual, leading the reported comparison on both, and says it was jointly trained across 20+ scientific domains with long-horizon agent RL.

AIIntern Large Models introduces Intern-S2-397B, a multimodal foundation model built for long-horizon scientific research and scientific agents. The post reports leading open-source results on IMO-Proof and AdvancedMathBench, and says the model reaches the level of Gemini 3.1 Pro on those tasks. It is now supported by vLLM and SGLang, with weights on Hugging Face and ModelScope and a chat demo available.

AIGPT Image 2.5 is now available in ChatGPT, and the post says it keeps consistency well while editing images. It also claims state-of-the-art results on all image leaderboards, and suggests users who saw faces shift in GPT Image 2 edits try again.

AIinclusionAI released Step-3.7-Flash-singprobe, an 8.13M-parameter probe that reuses Step-3.7-Flash hidden states to score query intent, response unsafety, and hallucination risk at every generated token. The probe adds less than 0.5% decode-time overhead and reports 0.9858 R-AUC and 0.9295 T-AUC on streaming safety benchmarks. It is supported through SGLang and vLLM integration branches and loads from Hugging Face by checkpoint ID.

AIinclusionAI has released Qwen3.5-397B-A17B-singprobe, an intrinsic streaming guardrail built on Qwen/Qwen3.5-397B-A17B that scores query intent, response unsafety, and hallucination risk at every generated token using the base model's hidden states. The probe has 8.13M parameters, taps layers 18, 38, and 58, and adds less than 0.5% decode-time overhead. Training code is available at inclusionAI/SingProbe, and the probe runs through SGLang or vLLM integration branches.

AIFireworks AI reports that DeepSeek-V4.1-Flash scores 74.34% pass@1 on DeepSWE at $0.430 per task, close to GPT-6-Astra's 74.12% at $6.524. On Terminal-Bench 2.1 it scores 86.5% against Astra's 87.5% at about 12x lower cost per task, while on HLE it trails Astra alone at 34.52% versus 50.40%. The post also reports that a combined oracle router reaches 54.80% on HLE, and that serverless and dedicated API access is available with US-hosted endpoints coming soon.

AISebastian Raschka's third "Reasoning from Scratch" video covers building a math verifier for evaluating language models and for later reinforcement learning with verifiable rewards (RLVR) training. The walkthrough covers extracting final answers from boxed outputs, normalizing them, checking mathematical equivalence, and running evaluation on the MATH-500 dataset.

AIMike Knoop argues intelligence can be measured as the ratio of a decision's quality to the optimal decision, capped at 100%. He says Astra is already 80% optimal on ARC v3 speedruns and identifies horizontal data acquisition and efficiency/cost as the most plausible near-term areas for RSI. Background from @mhmazur reports that GPT-6 Astra scored 100% on the 25 ARC-AGI-3 public games using 6,485 actions versus a human baseline of 17,135.

AIMike Knoop says he sees a path to an ARC-AGI-4 benchmark focused on open-ended invention, which he calls the gating capability between zero-sum automation and positive-sum innovation. He argues that coordinated slowdown efforts would likely apply to everyone, including open-source work, and cites chain of thought and the transformer as inventions that grew out of open science research. He concludes the research frontier must stay open to keep humanity on a positive-sum path.

AIEpoch AI's newsletter reports that Huawei is far behind Nvidia and is unlikely to catch up this decade due to export controls. It also finds official US GDP statistics understate growth by about 0.3 percentage points over the past year, and that GPT-6 Astra set new records on Epoch's evaluations, including the Epoch Capabilities Index.

Why it matters: The newsletter bundles several analyses of AI chips, GDP measurement, and benchmarks, so it helps readers scan the research agenda behind each finding.

AIAlberto Romero argues that AI solving Millennium Prize problems in 2026 threatens mathematics through success, not failure. He draws on Terence Tao's view that struggle shapes mathematicians, and that proof abundance without hard problems could leave fields depleted, like overplanted farmland.

AIInferact says vLLM now supports DeepSeek v4.1 Flash on day zero across H100, H200, B200, B300, GB200, and GB300 GPUs. SemiAnalysis independently verified the NVIDIA support, while the post notes AMD vLLM still does not work with the model. Serving recipes are available at recipes.vllm.ai.

AIDeepSeek released open weights for V4.1-Flash, which Baseten now offers through its Model APIs. The model has 552B total parameters, 8B active for prefill and 16B for decode, a 1M token context window, and text plus image input. Its Causal Encoder-Decoder design runs only the encoder during prefill and reuses a projected KV cache, and the source reports the global KV cache at a quarter of V4-Flash's memory.

Why it matters: The post explains how the CED architecture splits prefill and decode compute and cuts KV cache memory, which matters for coding agent costs.

AITailscale offers instant access to hundreds of models for any user in a secure tailnet through its customer-facing model router, Aperture. Aperture is built on Vercel's AI Gateway and offers zero data retention, zero markup with free BYOK, and cost and usage data on every response.

AIRedwood Research reports that better prompt templates raise chain-of-thought controllability scores on the CoTControl eval for open-source reasoning models by roughly 2-3x or more. For example, GPT-OSS-120B rose from 5.5% to 15% in the zero-shot setting. The author concludes that current CoT controllability numbers may underestimate what models can do, though the finding does not significantly undermine the view that current models probably cannot consistently evade CoT monitoring.

AIBAAI introduced AREX, a research agent built on a 122B-parameter mixture-of-experts model with 10B active parameters. It drafts candidate answers, checks each constraint, and revisits unresolved points rather than running one long search. The post says AREX performs on hard search benchmarks comparable to GPT-5.4.

AIAi2 released AstaBrief-8B-SFT, an 8B intermediate supervised fine-tuning checkpoint built on Qwen3-8B that turns a research question and retrieved literature excerpts into a cited report. On the ScholarQA-CS2 test set of 100 computer science questions, it scored an average of 83.7 versus 77.3 for base Qwen3-8B, with citation recall at 71.3 versus 64.6. The model is licensed under Apache 2.0 for research and educational use.

AIOpenAI's GPT-6 Astra is reported as the best-performing frontier model for antibody developability prediction in one benchmark, outperforming other frontier models tested on properties such as aggregation and stability. The post also says Astra built an interactive antibody visualization in about one hour.

AINVIDIA's Bryan Catanzaro welcomed the idea of models that understand how humans interact and thanked builders using Nemotron Ultra. Humansand introduced Persimmon, which it describes as the first large-scale model designed to realistically simulate how people talk and interact.

AIAmazon Science researchers propose that machine learning research agents avoid overfitting benchmarks despite years of iteration against the same tests. They attribute this to generalizable strategies being expressed compactly, leaving no room for memorization, while overfitting strategies fail to survive a compression bottleneck.

AICognition introduces SWE-2, a coding model post-trained from Kimi K3 that scores 50.0% on FrontierCode 1.1 Main, within one point of Fable 5.1 while costing 64% less. The post attributes the gains to an RL algorithm that trains all reasoning-effort levels in one run, with cost penalties tuned to the base model's Pareto frontier. SWE-2 is available starting today in Devin Desktop and CLI, with rollout to Devin Web and Fusion.

Why it matters: The post explains how the cost penalty and length-weighted baseline are derived, which helps readers judge the tradeoffs in coding model post-training.

AIMustafa Suleyman says Microsoft launched eight new models in two months, and five debuted at #1, including MAI-Transcribe-2 and MAI-Image 2.5, 2.6, and 2.6 Flash on the Artificial Analysis leaderboard. He says MAI-Cyber in MDASH ranked #1 on CyberGym while cutting costs by 50% versus other frontier models. He adds that MAI-Code-1.1-Flash, launched in GitHub Copilot four weeks ago, now accounts for a third of small-model traffic there.

AIAmazon Science researchers found that LLM research agents running benchmark hill-climbing rarely overfit, because their winning strategies can be compressed into prompts of about 32 tokens. A fresh reproducer agent with no access to the validation set matched the explorer's performance on most of eight datasets from that short prompt alone. The team also used the test to flag overfitting, since validation-specific gains did not survive compression.

AITencent Hunyuan has released AuK, an open-source foundation model for unified speech generation and editing that takes natural-language instructions and reference audio. It supports tasks including zero-shot TTS, timbre, style and emotion editing, denoising, and music separation. A companion AuK-Flash variant runs 4-step inference and is about 4.5 times faster under matched conditions, with code, weights, and a demo now available.

AIGeekbench 7 testing on the Snapdragon X2 Elite shows x86-64 binaries running through Windows 11's Prism translator lose substantial performance compared with native aarch64 execution. Binary translation roughly doubles executed instructions when running the x86-64 version, and every tested core, including Qualcomm's, takes a notable penalty. Even with that penalty, the Snapdragon X2 Elite's E-Cores outperform Neoverse N1 and its P-Cores outperform Neoverse N2.