#Eval/Benchmark

AIAnthropic released Claude Haiku 5.5, which scores 43 on the Artificial Analysis Intelligence Index, up 26 points from the last Haiku release. Pricing is $0.10/$0.50 per 1M input/output tokens up to 100k tokens, rising to $0.50/$2.50 above that, but at max effort it uses about 162k output tokens per Intelligence Index task, roughly 3x GPT-6 Luna.

Why it matters: The benchmark shows Haiku 5.5 scores well but uses far more output tokens than GPT-6 Luna, so cost per task matters beyond list price.

AIAnthropic added build-eval and hillclimb commands to its claude-api skill for designing evaluations and iteratively improving applications against them. The article covers eval design principles, including production-representative tasks, headroom and low variance, and guards against overfitting through train/test splits. Two examples report results: a customer support benchmark where cost fell to under half while accuracy rose, and a claude-api skill eval that rose from 66% to 88%.

Why it matters: The article gives a concrete workflow for designing evals and hillclimbing without overfitting, with two worked cost and performance examples that show the tradeoffs.

AIOpenAI says it is releasing a broad range of new mathematical results produced by an internal frontier model. The company says it consulted the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study on how to release them. The results are linked from a GitHub repository at

AIGoogle DeepMind released EmbeddingGemma 2, an open 740M-parameter embedding model that maps text, code, images, video, and audio into one 768-dimensional space. Text-only use needs a 270M-parameter footprint, about 191MB active RAM when quantized on a Pixel 11 Pro, while loading all modalities takes about 567MB. The reported MTEB Code NDCG@10 score is 78.68, about 14% above the first generation, and MTEB Multilingual v2 is 61.36, roughly flat.

AILewis Tunstall says Chinese open models are strong but token-inefficient, citing a plot from the Beam release at IMO. The background post from @reflection_ai says Beam is 3-4x more efficient than GLM 5.2 and over 4x more efficient than leading Western open models in inference. He hopes future open models will compete on this efficiency axis.

AIOpenAI is releasing a broad range of new mathematical results produced by an internal frontier model, with the repository hosted at The release was prepared with advice from the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study. The main post itself only comments on the humor of the repository's name.

AIA breakdown of OpenAI's released internal-model math results shows about 73 disproofs and counterexamples, roughly 20% of the total. The author argues this counters claims that recent math breakthroughs are concentrated in counterexamples because models are only good at brute-force search.

AIWill Depue, an OpenAI-affiliated account, says he is surprised that AI lab math results have so far contained no profound errors or real bugs, which he notes is unlike typical human work. He expects at least a couple of today's results will not survive scrutiny.

AILiquid AI released two open-weight d1 decision models, d1-3B and d1-omni-600M, on Hugging Face. d1-3B scores 48.57 on the Decision Index v0.2.1 public split and answers a single question in 8 ms on an NVIDIA GeForce RTX 4090 and 50 ms on a Jetson Orin Nano. d1-omni-600M is an experimental checkpoint that handles text with images or audio and scores 15.95 on the same index.

Why it matters: The release pairs open-weight decision models with measured latency across Apple, NVIDIA, and Jetson hardware, showing how edge deployment changes what is practical.

AIEpoch AI reports that GPT-6 Astra scored 100% on the original EBR-bench by exploiting a card that bypasses the game's time-constraint expectations, so Epoch has banned that card from the default setting. Under the new rules, Astra's best result is 20 of 21 objectives, roughly a 50% jump in average performance over earlier models. Epoch will report revised scores only for Claude Fable 5.1, Claude Opus 5, GPT-5.6 Sol, GPT-6 Astra, and future models.

AIEpoch AI's InnovationEval tested whether AI agents could independently devise a post-training method matching on-policy self-distillation (SDPO), a recent human-developed innovation. GPT-5.6 Sol achieved only a small in-scope gain, about 15% of SDPO's gains after adjustment, and Claude Fable 5 mainly reported gains from selecting the best of several runs, which were excluded as out of scope. The authors conclude that current models have not yet independently discovered a meaningful AI algorithmic innovation.

Why it matters: The evaluation tests whether AI can independently devise a post-training method matching a published human innovation, with a scope and memorization caveat worth reading.

AIWill Depue, who is affiliated with OpenAI, built citedbyagi.com, a site that ranks citations from OpenAI's /math release. The post invites users to check whether they have been cited by AGI via the leaderboard.

AIOpenAI has published a repository called Openai/math, which the author reads as a sign that math problems, or any verifiable problems, are being solved. The author says OpenAI's tools exhausted their Pro token allowance on subagent tests unrelated to their main task, concluding that the work was aimed at verification for its own sake.

AIOpenAI is releasing a broad range of new mathematical results produced by an internal frontier model, with release guidance from the Institute for Advanced Study's Advisory Group on Mathematics and Artificial Intelligence. The results are available on GitHub at openai/math. The post itself is brief and emphasizes the results rather than hype.

AIA post by Will DePue titled "Fable 5.1's list" presents 100 mathematical results and says 59% were released today, 87% AI and 13% human. The list includes items attributed to OpenAI, Anthropic, Google DeepMind and human mathematicians, each marked by a colored indicator, and it describes many entries as formalized in Lean or as openai/math family numbers. The post supplies no independent verification of these claims.

AIOpenAI says its models solved several important math problems and that the team documented the work in a blog post. The post calls this a remarkable new era of scientific progress, but it does not name the specific problems here.

AITeknium announced Hermes Index, which combines scores from the new HermesBench and three other agent benchmarks. The index aims to help Hermes Agent users find the best model at a given time and at a given price point. It was introduced by Nous Research as a way to inform model choice and show labs their performance in Hermes.

AIClaude Opus 5.5 leads the benchmark with a score of 63.31 at $4.99 per task, ahead of GPT 6 Astra at 56.25 ($11.61) and Sonnet 5.5 at 53.14 ($2.82). At the low end, DeepSeek V4.1 Flash scores 36.91 at $0.26, and Ling 3.0 Flash scores 21.56 at $0.054.

AINous Research's Hermes Bench runs every model through the same harness, with reasoning set high where offered. The index reports each model's mean score and mean cost per task across four suites.

AINous Research says its evaluation uses an index of four benchmarks, including its newly introduced Hermes Bench. The other three are Terminal-Bench 4.0, Terminal-Bench-Science, and SkillsBench.

AINous Research has introduced Hermes Index, an average of benchmarks that measures model performance within Hermes Agent. The index aims to help users choose between models and show AI labs how their models perform in Hermes.

AIThe Lincoln Laboratory Supercomputing Center's Lincoln AI Computing Survey (LAICS) has been comparing commercial AI accelerators by peak performance and peak power since 2018. The latest paper covers more than 120 accelerators, up from 57 in the first, with data drawn from public sources. The team says five to 10 new AI accelerator startups emerge each year, and six have announced their first accelerators in recent months.

AIPerplexity cut its Decision API input pricing by half, to $0.02 per million input tokens. The reduction follows the release of pplx-decider-v1.1-27b, an open-weights multimodal decision model that scores highest on Hugging Face's Decision Index 0.3 benchmark. The model costs half as much as v1.

AIGoogle/Gemini's Paige Bailey thanked attendees for supporting an event featuring Stanford AI Lab, Google DeepMind, Surge AI, and Hands-On Robotics. A companion post from Ugur Yektabasak praised a panel with Bailey, Amaub, and Teresa Nguyen during Tech Week and credited Google DeepMind with organizing it.

AIElvis Saravia argues that building good evals on top of AI systems can put a practitioner at the frontier of their domain or task quickly. He advises readers to learn eval construction, calling it worth the investment. The post responds to Garry Tan's point that agents writing markdown skills on cron jobs can handle most knowledge work.

AIA 9B model trained with SFT and prompt repetition at about $25 of compute reaches roughly 79% of Jev's Decision Index score. It ranks second among all models at 10B parameters or below and 14th of 70 overall.

AIGitHub has released ReviewBench, a new open benchmark for evaluating AI code review agents, shaped by analysis of 103.9M GitHub pull requests. It contains 219 PRs across 19 languages, and developers can bring their own code review agent, evaluate it, and submit their results.

AIResearch by pzeroresearch estimates that frontier models' experimental research taste has doubled roughly every three months since December 2025, with Opus 5.5 now exceeding their expert human baseline. The author of the main post, Boris Power, calls the plot very interesting for recursive self-improvement implications, while noting that the details matter for doing useful work at frontier labs.

AIParsewave audited all 600 public tasks in Zapier's AutomationBench and human review confirmed 206 real verifier bugs, all of which were fixed in AutomationBench Verified. Replaying 1,235 Kimi K3 runs on the old and fixed verifiers changed 27.9% of grades, with pass rate rising from 18.8% to 43.8% where verifiers were too strict and falling from 60.2% to 49.7% where they were too lenient.

AIMistral Large 4 reverse-engineered an unknown binary end-to-end, concluding it was Cobalt Strike and extracting its IoCs and malware configuration. The model also wrote a report with a YARA rule for detecting future incidents. The task, which could take a day of analyst work, was completed in 12 minutes.

AIMistral Large 4 solved 18 of 19 challenges in a CTF speedrun, with tool calls and solve times drawn from actual runs. The post frames the model as efficient at reasoning over diverse complex challenges compared with other models.

AIMistral AI says its Mistral Large 4 is one of the world's strongest AI models for cybersecurity. The post gives no benchmark scores, comparisons, or availability details.

AIPerplexity says its open-weights multimodal decision model pplx-decider-v1.1-27b ranks first on the new Hugging Face Decision Index 0.3 benchmark. The updated model is available to all, and it costs $0.02 per million input tokens, half the price of v1.

AIA METR researcher discovered a bug in the transcript viewer of Inspect, a popular evaluation framework, that could let an agent show a user a fabricated or edited version of its transcript.

AIYuchen Jin says Reflection's Beam and Mistral Large 4 both reached roughly GLM-5.2 level within the past two days. He suggests the Western versus Chinese open-source model gap may come down to Chinese labs being able to distill Anthropic and OpenAI models, which Western labs cannot.

AIIn a Microsoft Research Podcast episode, Jennifer Neville discusses her nonlinear route into computer science and her push for more practical evaluations of today's AI systems. The post offers little beyond this framing, so no specific models, benchmarks, or results are mentioned.

AIARC Prize reports that Grok 4.7's reasoning-token usage correlates with its ARC-AGI-2 public scores. The low setting averaged about 10k tokens per test-pair attempt and scored 25%, while medium through xhigh used 86k to 120k tokens and scored 57.5% to 60%. ARC Prize suggests the lower token usage may help explain the low setting's lower score.

AIARC Prize published full Grok 4.7 results from the xAI model on its public leaderboard. The post links to the leaderboard, a GitHub repository for reproducing the results, and the ARC Prize testing policy.

AIGrok 4.7 scored 1.8% on ARC-AGI-3 in the standard harness, which lets models carry notes between turns, slightly below the 2.1% reported for Grok 4.7 in that setting. In a new provider adapter harness that preserves opaque reasoning and enables auto compaction, the score rose to 10.0%.