Skip to contentSkip to stories

Updated

#Eval/Benchmark

Items with an AI score under 20 are hidden. Show low-relevance items

Oct 9

TodayOct 9Fri
  1. Artificial AnalysisOfficialAI score32

    HiDream-O1-Video-1.0 ranks #6 on Artificial Analysis image-to-video leaderboard

    AIHiDream-O1-Video-1.0 ranks #6 in Artificial Analysis's Image to Video with Audio leaderboard, just behind Dreamina Seedance 2.0 720p. HiDream says the model generates 1080p videos of 5 to 20 seconds with synchronized audio, priced at $5.80 per minute ($0.10 per second) on the HiHarness API. It is also available in vivago R1 Studio.

    GIF from @ArtificialAnlys's post
  2. Rohan PaulXAI score46

    Pine launches cloud computer for AI agents, reports 1/20 token cost

    AIPine has launched a cloud computer built for AI agents, which developers create through an SDK and give jobs in plain language. Running GPT-5.6 Luna, Pine reports about 1/20 the model-token cost of GPT-5.6 Sol with Codex on SaaS-Bench v1.1, scoring 78.3%, the highest in the published comparison. Pine also reports 1/26 the token cost of Opus 5 with Claude Code and 2 to 5 times faster speed in selected preliminary internal tests.

    Image from @rohanpaul_ai's post
  3. ClineOfficialAI score39

    Cline offers free access to Upstage's Solar Mini 4 model

    AICline is offering Solar Mini 4 free, a new 35B mixture-of-experts model from Korean lab Upstage with 3B active parameters. It has a 524K context window and runs at 208 tokens per second. Cline says it scores 24 on the AAII, the highest of any model at 3B active and within one point of Nemotron 3 Ultra, which uses 55B active.

  4. IdeogramOfficialAI score45

    Ideogram 4.5 keeps edited images intact across 30 consecutive edits

    AIArtificial Analysis ran 30 consecutive real estate staging edits through four image editing models, and Ideogram 4.5 kept most of each image unchanged while others drifted. Ideogram 4.5 and FLUX 3 left 95% or more of the image untouched on small edits, while GPT Image 2.5 Sunburst re-rendered most of the image and left only about a fifth unchanged. Nano Banana 2.1 kept its edits local but gradually darkened the rest of the image.

  5. Artificial AnalysisOfficialAI score34

    HeyGen Voice tops Artificial Analysis Controlled Voice TTS Arena leaderboard

    AIHeyGen Voice ranks first on the Artificial Analysis Controlled Voice TTS Arena leaderboard with an Elo of 1,201 across 1,468 appearances, ahead of Qwen-Audio-3.1-TTS-Plus at 1,182 and ElevenLabs' Eleven v4 Turbo at 1,166. On pronunciation robustness it scores 83.1%, ranking #10 of 29 models, and it is priced at $30 per 1M characters and processes 40 characters per second.

    GIF from @ArtificialAnlys's post
  6. Artificial AnalysisOfficialAI score22

    HeyGen Voice ranks first in Artificial Analysis assistant and customer service arenas

    AIHeyGen Voice ranks first in the Artificial Analysis Controlled Voice Arena for Assistants at 1,214 and Customer Service at 1,212. It ranks second in Knowledge Sharing at 1,158 and fourth in Entertainment at 1,178. By accent, it ranks first in English (UK) at 1,190 and second in English (US) at 1,206, behind Qwen-Audio-3.1-TTS-Plus at 1,220.

    Image from @ArtificialAnlys's post
  7. Artificial AnalysisOfficialAI score32

    HeyGen Voice scores 83.1% on pronunciation robustness benchmark

    AIHeyGen Voice scores 83.1% overall on Artificial Analysis's Pronunciation Robustness benchmark, ranking #10 of 29 models. The benchmark has human reviewers judge whether text-to-speech models pronounce challenging text correctly against pre-agreed accepted pronunciations. HeyGen Voice places #2 for preserving exact sequences at 83.3%, behind SpaceXAI TTS at 85.7%, and scores 77.0% on expanding shorthand, while Eleven v4 leads that category at 94.1%.

    Image from @ArtificialAnlys's post
  8. 🚨 AI News | TestingCatalogXAI score41

    Pine AI launches Pine Computer, a cloud runtime for agentic tasks

    AIPine AI launched Pine Computer, a cloud computer, harness, and runtime layer built for agentic tasks. On the publisher's SaaS-Bench v1.1, it posts a 78.3% checkpoint score against 74.3% for Opus 5 with Claude Code, but completes fewer whole tasks, 27.4% against 31.1%. Instead of simulating clicks and screenshots, it reads web pages as structured data, and access is through a private beta waitlist.

    Image from @testingcatalog's post
  9. OpenBMBOfficialAI score28

    MiniCPM5-2B runs at 37 tok/s on iPhone Air

    AIOpenBMB reports that its MiniCPM5-2B model runs at 37 tokens per second on an iPhone Air. The post presents this as evidence that small open multimodal models can run on mobile devices without a cloud GPU, with NobodyWho noting the model is available in its Chat app.

  10. MarkTechPostNewsAI score44

    Underdog Releases Saluki 27B, a 2-Bit Qwen3.8-27B That Beats the Original at Tool Calling

    AIUnderdog has released Saluki 27B under Apache 2.0, a 2-bit GGUF of Qwen3.8-27B that fits in 7.89 GB, versus 54 GB for the full BF16 model. On Underdog Bench, Saluki scores 88 against 84 for the full model, and it raises parallel tool-call accuracy to 42 from 35. It runs on stock llama.cpp, but math and reasoning drop sharply, with AIME 2025 at 79.2 versus 96.7.

  11. IThome · AINewsAI score55

    Odyssey-3 world model scores 66.1 on Physics-IQ Verified benchmark

    AIOdyssey announced the Odyssey-3 series of foundation world models, with Odyssey-3 Pro scoring 66.1 on the Physics-IQ Verified video-to-video benchmark, the highest recorded on that leaderboard. The series includes a standard version balancing physical accuracy and generation cost, and a Pro version with stronger physics prediction. The preview supports first-person and third-person navigation and lets users move the camera, take actions, or trigger events while the model predicts environmental changes in real time.

  12. vLLMOfficialAI score42

    vLLM Semantic Router team releases Decision 2.0 multi-question classification models

    AIThe vLLM Semantic Router team has released Decision 2.0, which answers multiple questions about one input in a single forward pass and outputs per-option probabilities. The post presents this as useful for routing and classification. A quoted post from Xunzhuo Liu says Decision 2.0 includes six open decision models ranging from 0.6B to 27B parameters, each topping same-size open models on the Jev Decision Index 0.3.

Oct 8

Oct 8Thu
  1. PandailyNewsAI score57

    Shanghai AI Lab Open-Sources Intern-Decision Small Models for Structured Decisions

    AIShanghai AI Lab has open-sourced Intern-Decision, a family of 0.8B, 2B and 4B parameter models that return structured decisions with probabilities instead of free text. The developers self-report that the 4B model averages 90.02% accuracy across seven test suites, ahead of a commercial reference model at 88.74%, with about 44 milliseconds of local latency on a single RTX 4090. Weights are on Hugging Face, and MetaX says the models run on its hardware from launch.

  2. Jerry LiuXAI score38

    LightOn OCR-3 now on OpenDocRouter, near Gemini 3.8 Flash at lower cost

    AILightOn OCR-3 is now available on OpenDocRouter at $0.28 per 1M input tokens and $1.40 per 1M output tokens, about $3.19 per 1k pages on ParseBench. On ParseBench, the author says it sits on the Pareto frontier for open-weight OCR models, with performance similar to Gemini 3.8 Flash low at roughly 45% lower price. It is described as decent at tables, workable for charts, and quite good at grounding.

    Image from @jerryjliu0's post
  3. Artificial AnalysisOfficialAI score38

    GPT-6 Sol (Daybreak Blue) tops Artificial Analysis Cyber Index

    AIGPT-6 Sol (Daybreak Blue, max) has been added to the Artificial Analysis Cyber Index as a trusted-access model and ranks #1 on the Index. Compared with the publicly available GPT-6 Sol, it shows its largest gains on CyberGym-E2E, the benchmark where the most safety refusals are observed.

    Image from @ArtificialAnlys's post
  4. Boris PowerXAI score46

    OpenAI's GPT-6.1-Sol leads new Arena Alignment Index for agents

    AIThe Arena Alignment Index, built from over 90K real-world agent sessions across 27 models, ranks OpenAI's GPT-6.1-Sol first with a score of 87.9, ahead of Claude-Opus-5.5 at 83.2 and Grok-4.7 at 82.7. GPT-6.1-Sol also posted the lowest observed rates across the index's three signals: 0.89% Unauthorized Action, 1.98% False Attribution, and 2.34% Deceptive Completion. The index's authors report that newer models consistently outperform their predecessors across all four labs, suggesting broad progress in agent safety.

  5. LiveKitOfficialAI score22

    LiveKit Simulations lets teams test voice agents before customers do

    AILiveKit is offering free access to its Simulations product through October, letting teams check what their agent can do and find gaps before deployment. The product also lets teams test any model against their own scenarios before switching models.

    Video from @livekit's post
  6. Alexander DoriaXAI score46

    LightOnOCR-3 claims state-of-the-art OCR performance under 1B parameters

    AILightOn has released LightOnOCR-3, a family of OCR models in 0.8B and 4B versions that it says lead benchmarks including OlmOCR-Bench and ParseBench, with the 0.8B model positioned as the sub-1B option. The models recognize text, handwriting, images, charts and document structure in one pass, process documents up to twice as fast as LightOnOCR-2, and are released under the Apache 2.0 license.

    Image from @Dorialexander's post
  7. 🚨 AI News | TestingCatalogXAI score49

    Odyssey launches Odyssey-3 world model with public research preview

    AIOdyssey has launched Odyssey-3, its most powerful foundation world model, with a public research preview. Odyssey-3 Pro scored 66.1 on Physics-IQ Verified video-to-video with best-of-8 sampling, the highest reported result. The model generates environments from prompts and predicts changes in real time as users move through scenes.

    Image from @testingcatalog's post
  8. elvisXAI score42

    Odyssey-3 Pro tops Physics-IQ Verified and shows robotic error recovery.

    AIOdyssey released Odyssey-3 Pro, which sets a new top score on Physics-IQ Verified, a benchmark where models continue videos of real physics experiments. In robotics, a robot arm with tens of hours of demonstrations recovered from a missed grasp, a behavior absent from those demos.

    Video from @omarsar0's post
  9. SantiagoXAI score46

    Odyssey 3 Pro world model tops Physics-IQ and goes live

    AIOdyssey 3 Pro, a world model, is now live as a research preview and ranks first on the Physics-IQ Verified video-to-video benchmark. The post says it can learn from visual observations and map that knowledge to physical controls for robots, cars, video games, and drones. Odyssey-3, the model launched alongside it, is described as free to try.

    Image from @svpino's post
  10. OdysseyOfficialAI score22

    Odyssey-3 Pro sets new Physics-IQ video-to-video benchmark record

    AIOdyssey-3 Pro achieved a score of 66.1 on Physics-IQ Verified's video-to-video benchmark, the highest reported score so far. Physics-IQ evaluates physical behavior across fluid dynamics, optics, solid mechanics, magnetism, and thermodynamics.

    Image from @odysseyml's post
  11. Arena.aiOfficialAI score55

    Arena raises $200M Series B and launches Alignment Index for AI agents

    AIArena announced a $200M Series B at a $3.1B valuation and released its Alignment Index, a benchmark measuring agent safety and alignment. The index is built from 90K+ real-world agent sessions across 27 models and tracks Unauthorized Action, False Attribution, and Deceptive Completion. OpenAI's GPT-6.1-Sol leads with a score of 87.9, ahead of Claude-Opus-5.5 at 83.2 and Grok-4.7 at 82.7.

    Video from @arena's post
  12. Understanding AI (Timothy B. Lee)BlogAI score67

    TypeSafe AI's Jev returns probabilities over fixed answers instead of text

    AITypeSafe AI released Jev, a model that answers yes/no, multiple-choice, or rating questions by outputting the estimated probability of each option. The author notes this design lets the model be served faster and more cheaply than LLMs and fits ordinary if-statement logic, and says he used it to flag spam comments on his blog in place of Gemini 3 Flash.

  13. Artificial Analysis ArticlesOfficialAI score62

    GPT-6 Sol Daybreak Blue leads the Artificial Analysis Cyber Index

    AIArtificial Analysis is adding trusted-access models to its Cyber Index, starting with GPT-6 Sol (Daybreak Blue, max), which is available only through OpenAI's Daybreak program. The model hits no safety blocks across the Index and scores 32 points higher overall than the publicly available GPT-6 Sol (max), with its largest gains on CyberGym-E2E.

    Why it matters: The source shows how safety refusals shape cyber benchmark scores, with the trusted-access model's gains concentrated on CyberGym-E2E, useful for comparing guarded and unguarded models.

Oct 2

Oct 2Fri
  1. Hugging Face BlogOfficialAI score70

    Ai2 open-sources AstaBrief 8B, a fast model for generating cited research reports

    AIAi2 released AstaBrief 8B, an open-weights model that turns a research question and retrieved literature excerpts into a cited report, along with its training data. The model runs as Fast mode in Asta, averaging 51.1 seconds per report versus 178.5 seconds for Thinking mode, about 3.5x faster. The post also describes filtering synthetic training data by citation density and building DPO pairs judged by two models that agreed.

    Why it matters: The post explains how supervised fine-tuning, preference data, and citation-density filtering were used to build a cited-report model, which is useful for teams training their own models.

Sep 29

Sep 29Tue
  1. BAAI · new models on Hugging FaceOfficialAI score62

    BAAI releases AREX-2, a 27B agent model for self-improving long-horizon tasks

    AIBAAI released AREX-2, a 27B-parameter long-horizon agent model that improves solutions over multiple test-time rounds by proposing, measuring, reflecting, and revising. It was trained on machine-learning and algorithmic-programming tasks with verifiable feedback, and the source reports that this self-improvement transfers to deep research. The model is Apache License 2.0 licensed and has a 262,144-token context length.

    Why it matters: The source compares AREX-2 against closed and open models on coding and deep-research benchmarks, showing how test-time self-improvement is measured across task types.