#Eval/Benchmark

AISantiago says physical consistency is the most important and hardest feature of a world model, and that many generated videos show objects defying gravity. He reports that Seedance 2.5 is currently the best among the evaluated world models. The post links to a physics evaluation benchmark in which eight video world models reached a top score of 57.76/100.

AIBoris Power, who owns OpenAI's account, praised the pace of progress on a result he called remarkable. The context from @0xdoug reports a PR merged and a tightened bound from κ = 2⁻¹⁸² to κ = 2⁻¹⁵, a community effort across contributors.

AIAnthropic released Claude Haiku 5.5, raising its OSWorld 2.1 score from 15.7% to 72.4% and supporting a 1 million token context window. The article notes Haiku 5.5 still scores 39.2% on Terminal-Bench 4.0 against Sonnet 5.5's 70.6%, and that prompts above 100,000 tokens are priced higher, so migration costs need to be measured on real workloads.

AIZyphra reports that patterns in model training emerge early enough to be useful and are inexpensive to measure. Using only 4,096 tokens, its placement and routing predictions come within roughly one percentage point of results obtained with 524,000 tokens.

AICohere is hosting a live webinar on the future of search and retrieval, covering Embed 5, Parse 5, and its new retrieval methodology, RCP-nDCG. The post is a livestream announcement and does not include details of the methodology or model performance.

AIBoris Power, who owns the OpenAI account posting this, responded to a third update on OpenAI problem #109, integer multiplication. The quoted update reports a tightened bound of κ > 2⁻¹⁸, improved from κ = 2⁻¹⁸², with an exact witness of 4.19 × 10⁻⁶.

AIPerplexity's pplx-decider v1.1 scored 643 of 669 on clinical decisions versus 628 for Jev, according to a quoted post. The quoted post says it costs 42% less per decision at about the same speed and that its weights can be downloaded and run inside hospitals.

AITypeSafe AI released Jev, a model that answers yes/no, multiple-choice, or rating questions by outputting the estimated probability of each option. The author notes this design lets the model be served faster and more cheaply than LLMs and fits ordinary if-statement logic, and says he used it to flag spam comments on his blog in place of Gemini 3 Flash.

AIStepFun has released Step 5 Preview, its flagship model for agentic work, built on a sparse Mixture-of-Experts architecture with 27B active and 600B total parameters. The source says it performs strongly in software engineering and professional tasks, but the feed supplied only an excerpt, so benchmark details are not available here.

AIJetBrains released Mellum2.1, a 12B mixture-of-experts model with 2.5B active parameters under the Apache 2.0 license, built for coding agents. Post-training shifted to reinforcement learning across thousands of environments and millions of sandboxed runs, and the model is available on Hugging Face. The source reports gains over Mellum2 on LiveCodeBench, AIME, GPQA Diamond, BFCL v4, IFEval, and SWE-bench Verified, and says it serves almost twice the tokens of Qwen3.5-9B under heavy load.

Why it matters: The post shows how reinforcement learning in real sandboxed environments changed a compact open model's repository work, with benchmark gains against Mellum2 and two peers.

AIAnthropic released Claude Haiku 5.5, its fastest and most affordable small model, at prices up to 90 percent lower for most prompts under 100,000 tokens. Artificial Analysis ranks it first among small-class models on its Intelligence Index with a score of 43, but it consumes about three times the output tokens per task that GPT-6 Luna needs at maximum effort.

AIUniPat AI's PaperBenchX benchmark found the strongest model, GPT-6 Astra, fully reproduced only 13.98% of 93 real research-paper tasks across 12 scientific fields. Reproduction was judged by regenerating outputs in an isolated environment, with 3,168 expert-verified scoring items. UniPat has open-sourced 12 test tasks and kept 81 tasks closed to preserve long-term evaluation validity.

AIGuizang (@op7418) says he posted news of Anthropic's price cut for Haiku 5.5 and was attacked by commenters who view the pricing as aimed at Chinese models. He compares Haiku 5.5 with DeepSeek-V4.1 flash and Zhipu's GLM 5.3 flash, finding it cheaper than both and scoring one point higher than GLM 5.3 flash on Terminal Bench 4.0 in AA's test.

AINVIDIA, Princeton University, and the University of Maryland introduced PivotOPD, an on-policy distillation method that teaches multi-turn LLM agents to recover from their most damaging early mistake. Tested on Qwen3-1.7B and Qwen3-8B students, it posts the best average against 13 baselines on ALFWorld, WebShop, and Search-based QA. It recovers from 72.7% of replayed pivotal mistakes, versus 20.3% for standard OPD, with no added inference cost.

AIAnthropic released Claude Haiku 5.5, priced the same as OpenAI's GPT-6 Luna, with a 1M-token context window. Artificial Analysis scored it 43 on its Intelligence Index, slightly ahead of GPT-6 Luna at 38, but it uses about 3x more output tokens at max effort.

AIA Hacker News post links to an OpenAI math repository history file and reports that OpenAI has withdrawn three mathematical results. The feed supplied only the title and a link, so the specific results, reasons for withdrawal, and dates are not stated here.

AIAnthropic's Claude Haiku 5.5, released alongside Claude Opus 5.5 and Claude Sonnet 5.5, is reported to lead GPT-6 Luna across benchmarks, with OpenRouter measuring roughly twice the token output speed. Anthropic says Haiku 5.5 is its cheapest, fastest, and most capable small model, costing about 75% less to run than Claude Haiku 4.5 on average. The post also notes some CodeX users are reportedly migrating to Claude Code.

AIPerplexity has released pplx-embed-v2-late, a pair of ColBERT-style multimodal embedding models in 0.6B and 9B sizes that retrieve text, images and rendered PDF pages in a shared embedding space. Both are available on Hugging Face under the MIT license, while a hosted API endpoint is planned but not yet live.

AIArtificial Analysis is adding trusted-access models to its Cyber Index, starting with GPT-6 Sol (Daybreak Blue, max), which is available only through OpenAI's Daybreak program. The model hits no safety blocks across the Index and scores 32 points higher overall than the publicly available GPT-6 Sol (max), with its largest gains on CyberGym-E2E.

Why it matters: The source shows how safety refusals shape cyber benchmark scores, with the trusted-access model's gains concentrated on CyberGym-E2E, useful for comparing guarded and unguarded models.

AIHarvey LAB-AA v1.1 adds hallucination checks that audit every model deliverable against task source documents, with material hallucinations zeroing a task's score. GPT-6 Astra averaged 0.03 material hallucinations per task across 120 tasks, while Gemini 3.8 Flash averaged 13.96. Harvey uses GPT-6 Sol (high) as the hallucination checker, separate from its three-judge rubric panel.

AIOpenAI published 722 math manuscripts covering 372 result groups in a new GitHub repository, openai/math, all produced by an unreleased internal model. The author describes the results as including a near-Riemann hypothesis claim pushed to 0.875, and notes that 25 Fields Medal winners criticized the company's approach to AI math research.

Why it matters: The piece traces how AI math results moved from benchmarks to open problems, offering context on verification and the mathematicians' pushback.

AIFrançois Chollet compares AI progress to human learning, noting that 1980s research found programming training improves coding but does not transfer to general reasoning. He argues general intelligence is a fundamental brain property rather than a trainable skill, since domain practice improves only that domain. The post is framed as background for his question whether AI's jagged frontier, driven by math and code via RLVR, reflects general capability or continued human-data bottlenecks.

AIOpenAI announced on X that it is releasing a series of new mathematical results produced by its internal frontier model. The material is reported as 722 manuscripts posted to GitHub, and the excerpt provides no further detail on the specific results.

AIFrench AI company Mistral has released Mistral Large 4, its new flagship model with 1 trillion total parameters. The company plans to make all of the model's weights open at the end of the month.

AIWaymo research found that even fully sober human drivers face nighttime fatal crash risk 3.1 to 3.9 times higher than daytime risk, pointing to systemic hazards beyond impairment. The study used an exposure reconstruction model across the 50 most populous U.S. urban areas, showing removing alcohol-involved drivers lowers the average urban fatal crash rate by 23%, from 1.42 to 1.10 per 100 million miles.

AIEpoch gave six models 11 real work tasks from its own operations, including graphic design, data insights, and research design, and graded outputs against employee standards. Fable 5.1 and GPT-6 Astra led on average task performance, reliably handling well-defined work such as coding and computational analysis. The report finds that all models still fail on open-ended judgment, including matching Epoch's standards, designing informative experiments, and generating diverse ideas, so the authors conclude AI cannot yet replace workers at Epoch.

Why it matters: The report separates well-defined task reliability from open-ended judgment failures, which benchmark scores on easily verifiable tasks would miss.

AIGoogle Developers Blog introduces AQuA, an ambient quality agent that runs in a customer's Google Cloud project and samples production sessions to find recurring agent failures. In a 32-session travel-concierge sweep, it verified six issues and traced two of them to specific prompt lines, and a replay after the fixes raised full-session passes from 5/32 to 13/32. The post notes that verification and diagnosis are model-based, and that the tool proposes edits without applying them.

Why it matters: The post walks through a concrete production workflow, from sweep and verification to a code-anchored fix and replay, that shows how to diagnose silent agent failures.

AIAndrew Curran notes two posts five hours apart as a gauge of how fast progress is moving. The linked update tightens the bound on OpenAI problem #109 (integer multiplication) to κ = 2⁻⁵⁹, down from OpenAI's original κ = 2⁻¹⁸².

AIAnthropic released Claude Haiku 5.5, which it calls the fastest, cheapest, and most capable Haiku model so far. On average it costs about 75% less to run than Haiku 4.5, with input at $0.10 and output at $0.50 per million tokens for requests up to 100,000 tokens. Anthropic also cut Sonnet 5.5's cache read price from $0.20 to $0.10 per million tokens, which it says lowers run costs by about 20% on many agent tasks.

AIThe main post is a short reply saying reports of a company's death have been greatly exaggerated, with no details about products or figures. The background post from @h_nilforoshan reports that OpenAI's Decisions API, billed as a "Jev-killer," was benchmarked against Jev for HiringCafe, which serves 2.5 million users. On the task of scoring job-description relevance from 1 to 10, the author reports OpenAI costing 2x more and performing 5-10% worse.

AINoam Brown says LLMs have crossed a threshold by surpassing top human experts on some research problems, a jump that makes the recent surge in math results feel sudden. He expects breakthroughs in other domains to follow as models keep improving, though capabilities remain jagged and often still weaker than humans.