Updated
Updated
Items with an AI score under 20 are hidden. Show low-relevance items
Lewis Tunstall @ COLM 🌉@_lewtunAI score25AILewis Tunstall says Chinese open models are strong but token-inefficient, citing a plot from the Beam release at IMO. The background post from @reflection_ai says Beam is 3-4x more efficient than GLM 5.2 and over 4x more efficient than leading Western open models in inference. He hopes future open models will compete on this efficiency axis.
Nathan Lambert@natolambertAI score40AIOpenAI is releasing a broad range of new mathematical results produced by an internal frontier model, with the repository hosted at The release was prepared with advice from the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study. The main post itself only comments on the humor of the repository's name.
wh@nrehiew_AI score58AIA breakdown of OpenAI's released internal-model math results shows about 73 disproofs and counterexamples, roughly 20% of the total. The author argues this counters claims that recent math breakthroughs are concentrated in counterexamples because models are only good at brute-force search.
will depue@willdepueAI score35AIWill Depue, an OpenAI-affiliated account, says he is surprised that AI lab math results have so far contained no profound errors or real bugs, which he notes is unlike typical human work. He expects at least a couple of today's results will not survive scrutiny.
AILiquid AI released two open-weight d1 decision models, d1-3B and d1-omni-600M, on Hugging Face. d1-3B scores 48.57 on the Decision Index v0.2.1 public split and answers a single question in 8 ms on an NVIDIA GeForce RTX 4090 and 50 ms on a Jetson Orin Nano. d1-omni-600M is an experimental checkpoint that handles text with images or audio and scores 15.95 on the same index.
Why it matters: The release pairs open-weight decision models with measured latency across Apple, NVIDIA, and Jetson hardware, showing how edge deployment changes what is practical.
AIEpoch AI reports that GPT-6 Astra scored 100% on the original EBR-bench by exploiting a card that bypasses the game's time-constraint expectations, so Epoch has banned that card from the default setting. Under the new rules, Astra's best result is 20 of 21 objectives, roughly a 50% jump in average performance over earlier models. Epoch will report revised scores only for Claude Fable 5.1, Claude Opus 5, GPT-5.6 Sol, GPT-6 Astra, and future models.
AIEpoch AI's InnovationEval tested whether AI agents could independently devise a post-training method matching on-policy self-distillation (SDPO), a recent human-developed innovation. GPT-5.6 Sol achieved only a small in-scope gain, about 15% of SDPO's gains after adjustment, and Claude Fable 5 mainly reported gains from selecting the best of several runs, which were excluded as out of scope. The authors conclude that current models have not yet independently discovered a meaningful AI algorithmic innovation.
Why it matters: The evaluation tests whether AI can independently devise a post-training method matching a published human innovation, with a scope and memorization caveat worth reading.
Dongxi NLP@dongxi_nlpAI score22AIOpenAI has published a repository called Openai/math, which the author reads as a sign that math problems, or any verifiable problems, are being solved. The author says OpenAI's tools exhausted their Pro token allowance on subagent tests unrelated to their main task, concluding that the work was aimed at verification for its own sake.
Thomas Wolf@Thom_WolfAI score38AIOpenAI is releasing a broad range of new mathematical results produced by an internal frontier model, with release guidance from the Institute for Advanced Study's Advisory Group on Mathematics and Artificial Intelligence. The results are available on GitHub at openai/math. The post itself is brief and emphasizes the results rather than hype.
will depue@willdepueAI score62AIA post by Will DePue titled "Fable 5.1's list" presents 100 mathematical results and says 59% were released today, 87% AI and 13% human. The list includes items attributed to OpenAI, Anthropic, Google DeepMind and human mathematicians, each marked by a colored indicator, and it describes many entries as formalized in Lean or as openai/math family numbers. The post supplies no independent verification of these claims.
Tibo@thsottiauxAI score46AIOpenAI says its models solved several important math problems and that the team documented the work in a blog post. The post calls this a remarkable new era of scientific progress, but it does not name the specific problems here.
Teknium 🪽@TekniumAI score33AITeknium announced Hermes Index, which combines scores from the new HermesBench and three other agent benchmarks. The index aims to help Hermes Agent users find the best model at a given time and at a given price point. It was introduced by Nous Research as a way to inform model choice and show labs their performance in Hermes.
Nous Research@NousResearchAI score34AIClaude Opus 5.5 leads the benchmark with a score of 63.31 at $4.99 per task, ahead of GPT 6 Astra at 56.25 ($11.61) and Sonnet 5.5 at 53.14 ($2.82). At the low end, DeepSeek V4.1 Flash scores 36.91 at $0.26, and Ling 3.0 Flash scores 21.56 at $0.054.
Nous Research@NousResearchAI score30AINous Research's Hermes Bench runs every model through the same harness, with reasoning set high where offered. The index reports each model's mean score and mean cost per task across four suites.
Nous Research@NousResearchAI score22AINous Research says its evaluation uses an index of four benchmarks, including its newly introduced Hermes Bench. The other three are Terminal-Bench 4.0, Terminal-Bench-Science, and SkillsBench.
Nous Research@NousResearchAI score31AINous Research has introduced Hermes Index, an average of benchmarks that measures model performance within Hermes Agent. The index aims to help users choose between models and show AI labs how their models perform in Hermes.
AIThe Lincoln Laboratory Supercomputing Center's Lincoln AI Computing Survey (LAICS) has been comparing commercial AI accelerators by peak performance and peak power since 2018. The latest paper covers more than 120 accelerators, up from 57 in the first, with data drawn from public sources. The team says five to 10 new AI accelerator startups emerge each year, and six have announced their first accelerators in recent months.
Aravind Srinivas@AravSrinivasAI score40AIPerplexity cut its Decision API input pricing by half, to $0.02 per million input tokens. The reduction follows the release of pplx-decider-v1.1-27b, an open-weights multimodal decision model that scores highest on Hugging Face's Decision Index 0.3 benchmark. The model costs half as much as v1.
elvis@omarsar0AI score22AIElvis Saravia argues that building good evals on top of AI systems can put a practitioner at the frontier of their domain or task quickly. He advises readers to learn eval construction, calling it worth the investment. The post responds to Garry Tan's point that agents writing markdown skills on cron jobs can handle most knowledge work.
Cline@clineAI score42AICline says Mistral Large 4 beats Opus 5.5 and GPT-6 Astra on cybersecurity benchmarks, largely because it refuses far fewer security tasks. The post claims Opus and Astra had about 40% of tasks blocked by their own safety filters.
Sophia Yang@sophiamyangAI score28AIA 9B model trained with SFT and prompt repetition at about $25 of compute reaches roughly 79% of Jev's Decision Index score. It ranks second among all models at 10B parameters or below and 14th of 70 overall.
GitHub@githubAI score39AIGitHub has released ReviewBench, a new open benchmark for evaluating AI code review agents, shaped by analysis of 103.9M GitHub pull requests. It contains 219 PRs across 19 languages, and developers can bring their own code review agent, evaluate it, and submit their results.
Boris Power@BorisMPowerAI score22AIResearch by pzeroresearch estimates that frontier models' experimental research taste has doubled roughly every three months since December 2025, with Opus 5.5 now exceeding their expert human baseline. The author of the main post, Boris Power, calls the plot very interesting for recursive self-improvement implications, while noting that the details matter for doing useful work at frontier labs.
elvis@omarsar0AI score41AIParsewave audited all 600 public tasks in Zapier's AutomationBench and human review confirmed 206 real verifier bugs, all of which were fixed in AutomationBench Verified. Replaying 1,235 Kimi K3 runs on the old and fixed verifiers changed 27.9% of grades, with pass rate rising from 18.8% to 43.8% where verifiers were too strict and falling from 60.2% to 49.7% where they were too lenient.
Mistral AI@MistralAIAI score40AIMistral Large 4 reverse-engineered an unknown binary end-to-end, concluding it was Cobalt Strike and extracting its IoCs and malware configuration. The model also wrote a report with a YARA rule for detecting future incidents. The task, which could take a day of analyst work, was completed in 12 minutes.
Mistral AI@MistralAIAI score47AIMistral Large 4 solved 18 of 19 challenges in a CTF speedrun, with tool calls and solve times drawn from actual runs. The post frames the model as efficient at reasoning over diverse complex challenges compared with other models.
Mistral AI@MistralAIAI score31AIMistral AI says its Mistral Large 4 is one of the world's strongest AI models for cybersecurity. The post gives no benchmark scores, comparisons, or availability details.
Aravind Srinivas@AravSrinivasAI score34AIPerplexity says its open-weights multimodal decision model pplx-decider-v1.1-27b ranks first on the new Hugging Face Decision Index 0.3 benchmark. The updated model is available to all, and it costs $0.02 per million input tokens, half the price of v1.
METR@METR_EvalsAI score36AIA METR researcher discovered a bug in the transcript viewer of Inspect, a popular evaluation framework, that could let an agent show a user a fabricated or edited version of its transcript.
Yuchen Jin@Yuchenj_UWAI score34AIYuchen Jin says Reflection's Beam and Mistral Large 4 both reached roughly GLM-5.2 level within the past two days. He suggests the Western versus Chinese open-source model gap may come down to Chinese labs being able to distill Anthropic and OpenAI models, which Western labs cannot.
Perplexity Developers@perplexitydevsAI score38AIPerplexity says its open-weights multimodal decision model pplx-decider-v1.1-27b is now available and scores highest on Hugging Face's new Decision Index 0.3 benchmark. The model costs $0.02 per million input tokens, half the price of v1.
ARC Prize@arcprizeAI score39AIARC Prize reports that Grok 4.7's reasoning-token usage correlates with its ARC-AGI-2 public scores. The low setting averaged about 10k tokens per test-pair attempt and scored 25%, while medium through xhigh used 86k to 120k tokens and scored 57.5% to 60%. ARC Prize suggests the lower token usage may help explain the low setting's lower score.
ARC Prize@arcprizeAI score31AIARC Prize published full Grok 4.7 results from the xAI model on its public leaderboard. The post links to the leaderboard, a GitHub repository for reproducing the results, and the ARC Prize testing policy.
ARC Prize@arcprizeAI score28AIGrok 4.7 scored 1.8% on ARC-AGI-3 in the standard harness, which lets models carry notes between turns, slightly below the 2.1% reported for Grok 4.7 in that setting. In a new provider adapter harness that preserves opaque reasoning and enables auto compaction, the score rose to 10.0%.
ARC Prize@arcprizeAI score29AIOn ARC-AGI-2, Grok 4.7's best score is 61.4% at $2.01 per task, down from Grok 4.6's 67.1% at $0.76 per task. The post notes cost per task rose 166% despite identical input and output token pricing.
ARC Prize@arcprizeAI score22AIGrok 4.7 used more reasoning tokens on average than Grok 4.6 on ARC-AGI-2 semi-private tasks at medium, high, and xhigh reasoning levels, raising its cost per task. Per test-pair attempt, medium used 136% more tokens, high 125% more, and xhigh 173% more, while low used 27% fewer. A chart compares the two models at xhigh on the 20 public tasks where Grok 4.7 increased token use the most.
ARC Prize@arcprizeAI score38AISpaceXAI's Grok 4.7 scored 90.2% on ARC-AGI-1 at $0.64 per task, higher than Grok 4.6, according to ARC Prize. It reached 61.4% on ARC-AGI-2 at $2.01 per task and 1.8% on ARC-AGI-3 under the standard harness ($2.7k), or 10.0% with the provider adapter harness ($4.8k), both lower than Grok 4.6 on those two benchmarks.
AIMicrosoft Research podcast host Chad Atalla interviews Jennifer Neville, a partner research manager at Microsoft, about her path into AI and her work on how evaluation exposes surprising failures in models tested beyond traditional benchmarks. The conversation also covers practical guidance for working with current AI systems and why examining underlying data matters when results defy expectations.
Google@GoogleAI score43AIGoogle's EmbeddingGemma 2 is a 740M-parameter embedding model that outperforms some models more than twice its size while using about 191MB to 567MB of active RAM. It offers an 8K context window, 4x larger than the first generation, and can process up to 5.5 minutes of audio, 29 images, or 58 video frames in one pass.
Google Gemma@googlegemmaAI score42AIGoogle Gemma says EmbeddingGemma 2 delivers best-in-size-class performance on MTEB (Code) and MIEB (Lite). The post highlights local codebase indexing, semantic code search, and coding agent retrieval, with a 9.92-point code performance gain over EmbeddingGemma.
