#Eval/Benchmark

AITencent has released a preview of its Hy Image 3.5 image generation model, which product teams across Yuanbao, WorkRally, Ima, and other services are helping refine through co-design. Tencent Cloud prices the model at USD 0.024 per 2K output image, and it supports text-to-image and image-to-image generation with up to five reference images. Tencent said an internal blind evaluation found it on par with ByteDance's Seedream 5.0 Pro and slightly better than Nano-Banana Pro and Qwen-Image-3.0 Pro.

AIMike Knoop argues that formal verification is becoming feasible and is important for security. He adds that it does not automatically build human understanding, which he calls an even bigger alignment problem. The post is framed as a reply to Boris Cherny's report that Claude Opus 5.5 helped formally verify the Claude Agent SDK in Lean, producing 16 bug-fix PRs.

AIAlibaba's Qwen-Image-2.1 is now the top open-source model in both the Image Edit Arena and Text-to-Image Arena, according to Arena. In the Image Edit Arena it scored 1367 points, placing #16 overall and just 3 points behind GPT-Image-1.5-high-fidelity at #15.

AIAnt Ling released Ling-3.0-flash-fin, a finance-specialized open-weight model with 124B total parameters and 5.1B activated, offering high intelligence density for its size. It scores 54.9% on Finance Agent v2 at $0.045 per task, and the free API is available for a limited time, with an fp4 quantized version for local use.

AITinker says an open LLM can serve a Jev-like interface that takes discrete options and returns fast probabilities, since next-token prediction is already a probabilistic classifier. A post by @ekzhang1 reports that a $5, 10-minute supervised fine-tuning run on Tinker improved Qwen3.6-35B-A3B's handling of Jev-style prompts, with +8% on GPQA Diamond and +12% on MMLU-Pro.

AIRedwood Research found that padding prompts with meaningless filler tokens improves GPT-6-Astra's no-reasoning answers on serial reasoning tasks, rising from about 10-20% to about 50% on 4-hop natural facts. Other tested models improved far less, and the authors argue this means Astra can perform cognition it does not verbalize in its chain of thought, making such monitoring harder.

AIFireworks Research released Ember-1, a specialized model built on Kimi K3 that it says delivers the same quality with 40% fewer tokens. Across five industry benchmarks, Ember-1 matched K3 max quality at a fraction of the cost, and in two customer A/B tests it used about 35% fewer tokens per task. It is available as a Research Preview on Serverless, and Fireworks is also launching training support for customized models.

Why it matters: The source gives benchmark and A/B results for cutting reasoning tokens while holding quality, which bears on cost planning for coding and agent workloads.

AISimon Willison has published a write-up on Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna. The post includes comparison grids of pelican drawings across the model families at different reasoning levels.

AIKarl's AI Watts (@aiwarts) tests GPT Sol, GPT Luna, and Claude Opus 5 head to head on 3D, web design, motion effects, and writing. The post presents the models' hands-on results across these tasks as a direct comparison.

AIMayank's Rigel, a 2.3B-parameter MoE (360M active) hybrid Mamba-2 model, was pretrained across H100, A100, V100 GPUs and TPU v5p/v6e on one codebase. The model lands within a few points of Llama-3.2-3B while using under 1% of its pretraining FLOPs. Tri Dao praised the work's engineering effort and the model's strength for its small size.

AILovable now offers Opus 5.5, which it says matches Opus 5's results while finishing builds in a third to half fewer steps. Internal benchmarks showed Opus 5.5 scoring 4 to 6% ahead of Opus 5 on verification discipline, with step reductions of 26% to 57% and input token reductions of 21% to 59% across tasks.

AIBlack Forest Labs released FLUX 3 Action, an open-weights 7B world action model that outputs robot joint commands from camera frames, robot state, and a text instruction. On the RoboLab-120 benchmark it reports 42.92% task success, ahead of Cosmos3-Nano-Policy at 36.8% and π0.5 at 28.0%. The model is fine-tuned on DROID, is distributed under the FLUX Kommunity License v.1.0, and runs in about 32 GB of GPU memory in bfloat16.

Why it matters: The model card gives a benchmark comparison, parameter counts, and an action contract, so readers can judge how it compares with existing robot policies.

AITypeSafe AI released Jev, a transformer-based System-1 model that outputs calibrated probabilistic decisions instead of generating tokens, returning answers in 70–500 ms at $0.042 per million input tokens. The model is built for typed Choice, Score, and yes/no questions inside software pipelines, and it is available to everyone without a waitlist, with $5 in starting credits. Vercel, Cloudflare, LangChain, and Langfuse have added Jev to their platforms.

AITencent Hunyuan introduced WebCraftBench, a benchmark that tests AI agents by using the live web app and scoring aesthetics, usability, and whether the original request was met. Coverage-guided exploration reaches parts of the app that agents otherwise miss. On 197 human-validated pairs, the benchmark matches human preference 85.3% of the time.

AIMETR's preliminary evaluation concludes that Claude Opus 5.5 likely gives slightly higher AI R&D productivity uplift than Fable 5.1 but is unlikely to fully automate AI R&D. The evaluation used five capability tasks over 10 business days of API access, and METR says Anthropic reviewed and edited the summary before sign-off.

Why it matters: The report separates two claims about AI R&D acceleration and discloses that Anthropic reviewed the summary, which helps readers weigh its independence and evidence.

AIStepFun's Step 5 Preview scores 44 on the Artificial Analysis Intelligence Index at about $0.72 per task, matching Kimi K3 (max) at roughly 2.8x lower cost. The source reports strong reasoning results, including 46% on Humanity's Last Exam, but places it behind Qwen3.8 Max and GLM-5.3 (max) on agentic evaluations. Open weights are planned for October 15.

AIKilo's post says Hacktron spent hours failing to exploit a known flaw in an old image library, then a working exploit of OpenAI came within three hours after Claude Opus 5 shipped. The post argues that teams cannot afford model lock-in as frontier models change daily.

AIFireworks AI launched the Specialized Intelligence Index, which compares open, closed, and specialized models on domain benchmarks in seven initial areas: healthcare, legal, cybersecurity, finance, customer support, productivity, and software.

AIFrançois Chollet described summer 2026 as a crazy period for AI in a brief post with no further specifics. The post is linked to ARC Prize 2026's ARC-AGI-3 Progress Prize, where $37,500 in prizes will go to top open-source solutions on September 30, with Tufa Labs, Lord Han Solo, and NVARC3 currently leading.

AIArena says Xiaomi's MiMo-V2.6-Pro debuted at about #10 overall on Code Arena: WebDev with a 1628-point AutoEval score, tying Claude Fable 5 (High). That is a 153-point gain over MiMo-V2.5-Pro's 1475, and it ranks about #3 among open-weights models under an MIT license. Arena notes the score is early, based on a reward model rather than live human votes, so rankings may shift as more votes arrive.

AIXiaomi MiMo released MiMo-V2.6-Distill-Qwen-9B, a 9B agentic model made by supervised fine-tuning Qwen3.5-9B on MiMo-generated data, as an open starting point for agentic reinforcement learning research. It scored 61.1 on SWE Verified, versus 60.0 for Qwen3.5-9B, and 44.6 on SWE Pro, versus 32.0. The checkpoint is served with SGLang and a MiMo chat template, and its SFT data totals 77.4B tokens.

AIMax Woolf reports that prompting agents to iteratively optimize code can make it faster than current state-of-the-art libraries. He says the blog post includes his prompts and benchmark results.

AIXiaomi released MiMo-V2.6-Flash-RL, an efficiency-balanced checkpoint in its MiMo-V2.6 series, on Hugging Face. The model is a sparse MoE with 309B total and 15B activated parameters, supports text, image, video, and audio input, and offers a 1M-token context. The technical report says it was trained with a single mixed reinforcement learning run across coding, agent, visual, and cybersecurity tasks.

Why it matters: The report pairs its benchmark tables with the RL training method, which helps readers judge how the checkpoint's scores relate to its training approach.

AIXiaomi MiMo released MiMo-V2.6-Pro-RL on Hugging Face, a sparse MoE model with 1.02T total and 42B activated parameters and a 1M-token context. The technical report says it accepts text, image, video, and audio, and was trained with a single mixed reinforcement learning run across coding, agent, visual, and cybersecurity tasks.

Why it matters: The report pairs a 1.02T-parameter MoE model with an RL-based self-improvement method, useful for judging how reinforcement learning is scaled in frontier open models.

AIResponding to a post by ScarletKc, howie.serious simply agrees with the claim that GPT-6 Astra struggles with open-ended, exploratory work that lacks a fixed correct answer. The main post is a one-word endorsement (), while the quoted post argues GPT models excel at verifiable, goal-defined tasks and that Claude Fable handles open-ended exploration better.

AIMatei Zaharia, a prominent AI researcher, said it is very cool that GEPA works on Jev. The post is a short endorsement, linking to background about a test in which GEPA optimized Jev's prompts for extracting suspected adverse drug effects from medical sentences.