Updated
Updated
Showing low-relevance items too. Hide low-relevance items
Liquid AI@liquidaiOfficialAI score42AILiquid AI and InSilicoMeds released LongevityBench, an aging benchmark with 17 tasks spanning clinical records, DNA methylation, transcriptomics, proteomics, and genetics. On several tasks, Liquid AI's compact LFMs outperformed every frontier model the team evaluated. The team plans to present the work to the longevity research community at ARDD this week.
Tencent Hy@TencentHunyuanOfficialAI score62AIResearchers from Tencent Hy, Fudan University, and Tsinghua University released ExplorationBench, a benchmark that tests whether AI systems can discover hidden rules in executable Alien World sandboxes. Across 10 frontier systems, getting feedback from experiments outperformed thinking alone, with the best run reaching 89.0% after four rounds. The authors note that rankings barely transfer between the two worlds, and the code is listed as coming soon.
ModelScope@ModelScope2022OfficialAI score46AIIndexTeam has released Index-Translate, a multilingual family covering text, speech, dubbing, and long-document translation across 150 languages. Its 9B model scores 0.8789 on FLORES, 0.8209 on instTrans, and 0.7387 on MEME, and the 2B and 9B models are released under Apache 2.0.
AIIssue 942 of The Sequence links Anthropic's Claude Opus 5.5, reported for the week of September 21–27, to DeepSeek's September 19 environments paper and a report of AI-assisted biological discovery. The newsletter argues that progress increasingly depends on the surrounding machinery that governs where a model acts, what it observes, and how its conclusions are checked.
AIHamel Husain reviewed Anthropic's new build_eval and hill-climb commands in the claude-api plugin for Claude Code, finding it useful for discovering issues like human handoff, formatting, and voice agent problems. He criticized it for pushing evaluator creation before data review, asking for label validation in Markdown files, and bundling four failure checks into one broad call-transfer evaluator. Husain says he would hold off on using it for now.
ModelScope@ModelScope2022OfficialAI score62AIInSpatio-World 1.5 from InSpatio_AI turns a single image, four images, a panorama, or a video into a navigable scene with wide viewpoint changes. The 1.3B model scores 68.72 on WorldScore-Dynamic, ranking first among evaluated real-time and interactive methods, with speeds up to 24 FPS. The post says the code is released under Apache 2.0 and that dependencies keep their own licenses.
AIKorean AI lab Upstage has released Solar Mini 4, a proprietary reasoning model that scores 24 on the Artificial Analysis Intelligence Index with 35B total and 3B active parameters. It is priced at $0.10/$0.40 per 1M input/output tokens and has a 1M-token context window, but averages 7.1 minutes per task due to heavy output token use. Its weights are not released, and its size cannot be independently verified.
AIArtificial Analysis reports that Google's Gemini 4 Argon scores 53 on its Intelligence Index with high reasoning, matching GPT-6 Astra (max) and one point ahead of GPT-6.1 Sol (max). At the current 50% launch discount, its cost per task is $1.99, about 60% of GPT-6 Astra's $3.26, but the discount's end date is unconfirmed and standard pricing would raise it to $3.98. The model is being rolled out to selected users and is not publicly available.
Why it matters: The benchmark compares Gemini 4 Argon's cost per task and hallucination rate with GPT-6 Astra, showing where its value depends on a temporary 50% discount.
Jerry Liu@jerryjliu0XAI score20AIJerry Liu says Jev, a System One model, outperformed other open-source classifiers and document-specific models on orientation detection, language detection, classification, and splitting. The benchmark measured accuracy, cost, and latency across these fast document decisions, with Jev leading most comparisons. The benchmark code is available in the run-llama/jev_vs_oss repository.
Jerry Liu@jerryjliu0XAI score22AIJerry Liu benchmarked gpt-6.1 sol on document OCR tasks and found a sizable increase in table parsing and reading order over gpt-6 sol from a week earlier. Its table parsing is similar to gpt-6 astra. He noted frontier models still cost roughly an order of magnitude more than cost-effective document parsing solutions, leaving room to improve the premium end above 1c per page.
AIHugging Face released the Open TTS Leaderboard, which evaluates open-source text-to-speech models using objective metrics instead of arena-style human votes. It measures intelligibility via WER and CER using Qwen3 ASR, speed via RTFx and time-to-first-audio on an H200 GPU, and speaker similarity via WavLM embeddings. The leaderboard covers multilingual results and voice cloning, and it is intended to complement, not replace, human preference rankings.
AIApple researchers systematically tested LLM conditioning methods and found efficient activation steering often degrades fluency. Steering is far less effective on instruction-tuned models than base models, while prompting and full supervised fine-tuning work for concept injection but are weaker at concept removal. Cheap textual metrics correlate highly with costly LLM-as-judge scores.
Prime Intellect@PrimeIntellectOfficialAI score20AIPrime Intellect says it will be among the first to deploy on NVIDIA's new Vera CPU, after earlier access to benchmark sandboxes on Vera in March. The company plans to use Vera's dynamic memory latency for agentic workloads, which it describes as ideal for Prime Sandboxes.
AMD@AMDOfficialAI score18AIIn Signal65 RAG testing hosted via TensorWave, AMD Instinct MI355X delivered about 42% lower p99 latency. It also supported roughly 2x the concurrent-user headroom before breaching the evaluated SLA.
Liquid AI@liquidaiOfficialAI score32AILiquid AI announced d1, its first decision model, which it says is the first to outperform Jev on Hugging Face's Decision Index. The company claims d1 wins on multilingual evals, resists prompt injection better, handles longer inputs more effectively, and is built for fast, structured decision-making in software environments. It is available via the Liquid API at console.liquid.ai, with OpenRouter availability coming soon.
Noam Brown@polynoamialXAI score25AINoam Brown praised OpenAI for presenting model evaluations as intelligence plotted against cost, arguing that cost should be part of how intelligence is measured. OpenAI's linked context says GPT-6.1 Sol delivers near-Astra intelligence at one-fifth the price and is the most cost-efficient model for its performance available today.
François Chollet@fcholletXAI score12AIFrançois Chollet suggests a meta-benchmark for human-level generality: an AI system should be able to pass ARC-AGI-(n+1) immediately upon its release. A quoted reply from @dylantmoore adds that such a test would presumably not hold up forever.
AIBAAI released AREX-2, a 27B-parameter long-horizon agent model that improves solutions over multiple test-time rounds by proposing, measuring, reflecting, and revising. It was trained on machine-learning and algorithmic-programming tasks with verifiable feedback, and the source reports that this self-improvement transfers to deep research. The model is Apache License 2.0 licensed and has a 262,144-token context length.
Why it matters: The source compares AREX-2 against closed and open models on coding and deep-research benchmarks, showing how test-time self-improvement is measured across task types.
Cognition@cognitionOfficialAI score38AIAt low effort, GPT-6.1 Sol scores 58.1% at $0.21 per task, up from 50.5% for GPT-6 Sol at the same setting. That makes it the highest-scoring model under $0.30 per task on the leaderboard.
Cognition@cognitionOfficialAI score40AIGPT-6.1 Sol is now available in Devin, scoring 60.4% on FrontierCode 1.1, close to GPT-6 Sol's 60.7%. At medium reasoning effort it costs $0.31 per task, 81% less than GPT-6 Sol at max effort.
OpenAI@OpenAIOfficialAI score37AIOpenAI reports that GPT-6.1 Sol shows major alignment improvements over GPT-6 Sol in its evaluations, moving closer to GPT-6 Astra. The model is more transparent about its limitations and more reliable at respecting user intent and safety constraints.
Jerry Liu@jerryjliu0XAI score22AIJerry Liu hosted a dinner with Snorkel's Vincent Sun on evals and RL environments, a topic shaped by models rapidly saturating benchmarks. The conversation highlighted that building fair RL environments is hard, since failures are difficult to attribute to input, harness, or reward model, and that long-horizon evals spanning weeks or months remain very difficult. The post also noted that regulated industries still require human-in-the-loop review because 80% accuracy is not sufficient.
AIReplit explains how its Agent lets the core model pick subagent tier and effort mid-task rather than relying on an external router. On DeepSWE and Terminal-Bench, Replit Agent scored 72% at $2.11 per task and 49% at $2.53 per task, beating a single long-lived worker sidekick setup by 11 and 16 points. The company says Astra on its own scores higher only at more than twice the cost.
Why it matters: The post gives a concrete harness design with benchmark cost-score comparisons, helping builders weigh delegation strategies against routers and single-worker setups.
Sebastian Raschka@rasbtXAI score22AISebastian Raschka has published a comprehensive write-up on language models for text classification, covering RNNs, CNNs, transformers, and calibration. It is presented as a visual guide with hands-on experiments on accuracy and efficiency.
ModelScope@ModelScope2022OfficialAI score54AIModelScope announced IQuest-Q1, a 320B MoE model with 15B active parameters and a 512K context window for agentic coding. The post reports scores of 84.5 on CyberGym, 83.2 on Terminal-Bench 2.1, 64.6 on DeepSWE v1.1, and 63.0 on NL2Repo, and says weights are released under the IQuest-Q1 License.
ModelScope@ModelScope2022OfficialAI score44AIShanghai AI Laboratory's Intern-Decision family of 0.8B, 2B, and 4B multimodal models averages 79.38, 84.68, and 90.02 across seven decision benchmarks. Intern-Decision-4B scores 88.74, surpassing Jev while achieving better probability calibration. Reported mean latency is 33.98, 33.28, and 44.16 ms, versus 109.70 ms for Jev in the same local HF setup.
AIInternLM's AutoVerifier, built on Qwen3_5MoeForConditionalGeneration with about 68 GiB of weights across 40 safetensors shards, evaluates natural-language mathematical proofs, explains errors, and identifies the earliest incorrect step. It serves as the automatic grader for AdvancedMathBench's ProverBench, which accepts a proof only when all eight judgments report -1. The model is a learned grader rather than a formal proof checker and can make errors.
Thomas Wolf@Thom_WolfXAI score38AIThomas Wolf says the sharp decline in cheating by the latest Opus models is most likely due to evaluation awareness, meaning the models may recognize that the benchmark tests for cheating. If so, the benchmark no longer measures the models' natural tendency to cheat.
Thomas Wolf@Thom_WolfXAI score29AIThomas Wolf, owner of the Hugging Face account, posted the single word "impressive" in response to a quoted post. The quoted post reports a new NanoGPT training record of 39.9s, down 27.7s from the prior 67.6s, achieved through per-flop optimizations such as sampled softmax and sparse updates.
Matei Zaharia@matei_zahariaXAI score36AIMatei Zaharia said autoresearch produced strong results that are being integrated into a model serving stack. The post gives no specific figures, benchmarks, or product names. Background context from a related post says Databricks ranked #1 on NVIDIA's SOL-ExecBench kernel leaderboard across all four tracks using agents.
AIArtificial Analysis has open-sourced AA-AgentPerf-Local, a tool that replays recorded agent trajectories to measure inference speed on laptops and workstations. Initial results cover NVIDIA DGX Spark, NVIDIA GeForce RTX 5090, AMD Ryzen AI Halo, and MacBook Pro M5 Pro, with the RTX 5090 fastest for models that fit its 32 GB. The source states the tool and leaderboard will expand to more hardware, frameworks, and models.
Why it matters: The source gives per-system completion times and memory bandwidth figures, letting readers compare local hardware for running agentic workloads.
AIArtificial Analysis reports that GPT-6.1 Sol replaces GPT-6 Sol after seven days and scores 1 point below GPT-6 Astra on the Intelligence Index. At max effort it costs $0.72 per Intelligence Index task, compared with $3.26 for GPT-6 Astra and $1.05 for GPT-6 Sol. Its pricing matches GPT-6 Sol at $2/$10 per million input/output tokens, but it uses about 10-30% more output tokens.
Why it matters: The source compares GPT-6.1 Sol against GPT-6 Sol, GPT-5.6 Sol, and GPT-6 Astra on cost per task and token use, helping readers weigh performance against price.
AIAnthropic reports that Zhipu AI's GLM-5.3 can autonomously build end-to-end cyber exploits and is released without meaningful safeguards against misuse. In its simulated tests, attackers bypassed the model's safeguards 64% to 100% of the time using simple techniques, while the same attacks failed against safeguarded Claude models. Anthropic also cites an NIST CAISI assessment calling GLM-5.3 the most cyber-capable open-weight model released to date.
Why it matters: The report shows how open-weight safeguards fail under simple bypasses, offering concrete test figures for judging misuse risk in released models.
ModelScope@ModelScope2022OfficialAI score44AIAudio8 ASR Infinite transcribes Chinese and English audio of unlimited length using a rolling KV Cache that avoids accumulated drift. At a 480 ms delay, it reports 1.75 CER on AISHELL-1, 2.89 on AISHELL-4, and 3.04/6.81 WER on LibriSpeech test-clean/test-other. The preview release is under Apache 2.0, with deployment through an adapted vLLM stack.
Alexander Doria@DorialexanderXAI score14AIAlexander Doria says high parameter capacity still matters for harder document processing, running Astra for initial annotation and Gemma 4 31B for finetuning. Yifei Hu reports that gpt-6-sol improved over last week's version on domain-specific document parsing but remains far behind gpt-6-astra, with the benchmark itself built using Astra.
Ali Ghodsi@alighodsiXAI score62AIDatabricks tested recent AI models across 2,400 engineers and found Opus 5.5 offers the highest quality mid-tier performance, with about 20% lower same-task costs than Opus 4.8. The company is now encouraging Opus 5.5 as a default model for coding, and reports that GPT-6 Luna is at least 20 times cheaper per task than Opus 5.5, roughly matching Opus 4.6 on one difficult evaluation suite. The Luna findings are preliminary.
Felix Rieseberg@felixriesebergXAI score58AIAnthropic launched Sonnet 5.5, which the post says is smarter and more tasteful than Sonnet 5. It is positioned for work that does not need the extra capability of Opus or Fable.
Replicate@replicateOfficialAI score28AIReplicate has added P-Video-2-Pro, the latest video model from Pruna AI, which sits on the edge of the preference-speed and preference-price Pareto frontiers. Design Arena ranks its Quality and Speed variants tied for #2 on the Image to Video leaderboard with an Elo of 1325, with the Quality version generating in 8.0 seconds and the Speed version in 4.5 seconds.
AIVespper, a Y Combinator F24 startup, launches a DOCX MCP that lets agents edit Word files through HTML that a trained reconciler converts back to OOXML. On its 279-task internal benchmark, the company reports its MCP is 2.7–2.9x cheaper and 2.7–3.5x faster than Anthropic's DOCX skill, with higher pass rates. The post also lists current limitations, including no comment creation or reply, no image or video attachment, and no support for latent styles.
Fireworks@FireworksAI_HQOfficialAI score22AINormal Factory joins the Specialized Intelligence Index with CAD Arena, which tests whether AI agents can turn engineering drawings into accurate, editable CAD parts. The benchmark evaluates agents across five CAD platforms, extending the SII into engineering design.
