#Eval/Benchmark

AIArtificial Analysis shares an AA-Video-T2V v2.0 text-to-video prompt, a 10-second product-page scene of a presenter revealing a volcano-shaped mist diffuser. The prompt specifies a shift from warm daylight to dim evening lamplight, fixed close-up camera angles, and consistent presenter and product appearance across shots.

AIArtificial Analysis reports that Grok Imagine Video 1.5 Lite comes closest to the frontier on AA-Video-T2V v2.0 in Multi-Scene & Narrative, Lighting & Materials, and Text Rendering. It is furthest behind in Dialogue & Lip Sync and Human Anatomy. Compared with Grok Imagine Video 1.5, Lite matches it in Physics and trails on the other nine capabilities, by the least in Multi-Scene & Narrative.

AIAmong 12 models on AA-Video-T2V-Silent v2.0, Grok Imagine Video 1.5 Lite is the only one that is both fastest and highest quality, with no model beating it on both measures. It generates a 10-second 1080p clip in a median of 60.5 seconds. Kling 3.0 1080p (Pro) scores slightly higher but takes 94 seconds for a 5-second clip, while Vidu Q3 Turbo is 9 seconds faster on a 5-second 720p clip yet scores well below it.

AIGrok Imagine Video 1.5 Lite ranks #17 on AA-Video-T2V v2.0, two places above Google's Veo 3.1. At 1080p with audio, it costs $0.14 per second versus $0.40 per second for Veo 3.1. Compared with Grok Imagine Video 1.5, Lite is 44% cheaper at 1080p but ranks six places lower.

AISpaceXAI's Grok Imagine Video 1.5 Lite ranks #17 on both AA-Video-T2V v2.0 leaderboards, ahead of Google's Veo 3.1 at about a third of its price. It is the fastest model at its quality level in Artificial Analysis benchmarks, with a median of 60.5 seconds for a 10-second 1080p clip, and it costs $0.14 per second at 1080p, 56% of Grok Imagine Video 1.5's $0.25 per second.

AIArena's account of its continued work with The House Fund is a short acknowledgment with no product, figures, or new details. Read alongside the linked background, it sits in the context of Arena's $200M Series B at a $3.1B valuation and its Alignment Index for frontier models.

AISalesforce Ventures announced an investment in Arena's Series B round, with Arena described as a neutral platform where real users vote on anonymous model outputs across text, coding, vision, image generation, and agentic tasks. Labs and builders use its leaderboards to guide training and purchasing decisions.

AIArena announced a $200 million Series B at a $3.1 billion valuation and launched the Arena AI Alignment Index, a new measure of how AI agents behave in the real world. The main post is a thank-you from Felicis, a backer of Arena since its seed round, and the company's origins as a UC Berkeley research project.

AIArena has released its Alignment Index, a ranking that tracks AI models for unauthorized actions, deceptive completions, and false attribution of intent. The full technical report and rankings are available on Arena's blog and leaderboard site.

AIEpoch AI reports that GPT-6.1 Sol's latency behavior on long contexts suggests a change in how the model handles them, though not conclusively a new architecture. The post is a follow-up to its earlier report on latency scaling in frontier models.

AIEpoch AI reports that OpenAI halved GPT-6.1 Sol's cached-input price compared with GPT-6 Sol. Its measurements also show the model handles long prompts faster. The post raises, but does not confirm, a new architecture for GPT-6.1 Sol.

AISara Hooker announced a new blog post on adaptive checklists from Adaption AI, which she says capture current model capabilities and domain-specific requirements better than traditional checklists. She claims these adaptive checklists achieve this without requiring human supervision.

AICoreWeave is positioning its AI cloud around an open development loop, connecting training, inference and evaluation through its newly announced CoreWeave Forge platform. Chief marketing officer Jean English said the company wants production learnings to improve models and agents and that the loop should work across different models, frameworks and clouds. She argued that competitive differentiation extends beyond GPUs to partner tooling, infrastructure and APIs.

AIAnthropic's Claude Haiku 5.5 reached 1587 points on the Arena WebDev leaderboard, a 257-point gain over Claude Haiku 4.5's 1330. The post says it also improved significantly across all categories.

AIClaude Haiku 5.5 (High) debuted at #30 with 1587 points in Code Arena: WebDev, just outside the Pareto frontier. It matches GPT‑6 Luna's price at $0.10/$0.50 per 1M input/output tokens while scoring six points higher. Arena says it costs 90% less than Haiku 4.5 for +257 points, and 95% less than Sonnet 5.5 (High) for -128 points.

AIArtificial Analysis has released full evaluation results for its Cyber Index, which the post links at artificialanalysis.ai. The post itself gives no models, scores, or other figures, so the details are only available through the linked page.

AIArtificial Analysis has formed the Cyber Index Alliance to set a new standard for evaluating how AI models perform on enterprise cyber defense tasks. Current members are Collinear, IBM, NVIDIA, and Vercel, and partners contribute expert input on the Index design and implementation, plus datasets and external research. Organizations interested in joining can contact cyber@artificialanalysis.ai.

AIGPT-6 Sol (Daybreak Blue, max) has been added to the Artificial Analysis Cyber Index as a trusted-access model and ranks #1 on the Index. Compared with the publicly available GPT-6 Sol, it shows its largest gains on CyberGym-E2E, the benchmark where the most safety refusals are observed.

AIGPT-6 Sol (Daybreak Blue, max) ranks first on the Artificial Analysis Cyber Index at a Cost per Task of $1.77. That is significantly cheaper than other leading models, including Grok 4.7 (xhigh) at $11.67 per task.

AIArtificial Analysis added trusted-access models to its Cyber Index, and GPT-6 Sol (Daybreak Blue, max) now ranks first. The model is available only through OpenAI's Daybreak program and records no safety blocks across the Index. Its overall score is 32 points higher than the publicly available GPT-6 Sol (max), at a cost of $1.77 per task versus $11.67 for Grok 4.7 (xhigh).

AIEpoch AI estimates the AI chips shipped through 2027 could support about 30 to 170 million concurrent frontier-model agents, or nearly 2 billion with cheaper models. Its researchers find the cost of a fixed level of AI performance has fallen about 47% per quarter over the past three years. The newsletter also reports China's semiconductor supply-chain exposure is 2.7 times that of the US.

AIGoogle Research is presenting ContinuousBench, a standardized benchmark for measuring knowledge transfer and data contamination in differentially private (DP) synthetic data generation. Alex Bie is giving an encore presentation at the Google booth #107 at COLM 2026 today, at 2:00 PM PT.

AIGoogle proposed FlowAgent, a ReAct-style agent that generates and validates fixes for pre-submit test failures and shows them in its code review tools. Two abstention filters, before and after execution, suppress weak suggestions; in a manual review of 195 real failures, 67.18% of fixes were correct. After the Google-wide launch, it suggested fixes on 295,508 changes, with developers previewing 65,069 and applying 28,554.

AIArtificial Analysis and Harvey released LAB-AA v1.1, which credits a legal task only when deliverables pass every rubric criterion with no material hallucinations. Grok 4.7 (xhigh) leads at 9.4%, ahead of Muse Spark 1.3 (max) at 8.9% and GPT-6 Astra (max) at 8.6%, while over 60% of otherwise passing results contained a material hallucination. The sharper reordering appears in the hallucination counts, where GPT-6 Astra averages 0.03 material hallucinations per task against 13.96 for Gemini 3.8 Flash (high).

AIParticipants will use a dataset to build AI models, evaluated through a series of evals ranging from general benchmarks to more complex tasks. Top teams will be shortlisted and have their experimental hypotheses tested in Prima Mente's wet lab.

AIGoodfire and PrimaMente report discovering a new class of Alzheimer's biomarkers. They are now opening a global AI competition to find new treatments, called the Alzheimer's Translation Challenge, led by PrimaMente and AlzData.

AIThe Arena Alignment Index, built from over 90K real-world agent sessions across 27 models, ranks OpenAI's GPT-6.1-Sol first with a score of 87.9, ahead of Claude-Opus-5.5 at 83.2 and Grok-4.7 at 82.7. GPT-6.1-Sol also posted the lowest observed rates across the index's three signals: 0.89% Unauthorized Action, 1.98% False Attribution, and 2.34% Deceptive Completion. The index's authors report that newer models consistently outperform their predecessors across all four labs, suggesting broad progress in agent safety.

AIArtificial Analysis congratulated the launch of the ninth annual State of AI report, noting its benchmarks were featured throughout. The report, by Nathan Benaich, covers AI building better AI, world models, physical AI, geopolitics, cyber defense, and 12-month predictions.

AIArena has secured a $200M Series B, with Lightspeed doubling down on its investment. The company is also launching the Alignment Index, which measures how closely AI behavior aligns with human values in real-world settings. Arena reports annualized revenue above $100M since its Series A, with millions of people helping evaluate frontier models through real-world use.

AIJohn Groetzinger, writing in a personal capacity rather than for Cisco, argues that enterprise skills need packaging, evaluation, syncing, and distribution rather than scattered markdown files. He describes using skills to make cheaper models viable, converting curated TAC knowledge-base articles into maintained skills, and rolling out an eval framework across teams. He also describes syncing a repository README to Confluence with a deterministic script.

AILiveKit is offering free access to its Simulations product through October, letting teams check what their agent can do and find gaps before deployment. The product also lets teams test any model against their own scenarios before switching models.

AIArtificial Analysis has released Harvey LAB-AA, an evaluation built on Harvey's LAB dataset and developed in collaboration with Harvey. Full results are published on the Artificial Analysis evaluations page, alongside Harvey's commentary on the benchmark and human expert preferences.

AIArtificial Analysis reports that generating more output tokens does not necessarily yield a higher score. GPT-6 Astra (max) scored 8.6% using about 81k output tokens per task, while Grok 4.7 (xhigh) used roughly 180k yet scored lower. Three Claude models produced the most output tokens, about 202k to 562k per task, but scored between 2.8% and 6.4%.

AIAmong models with a Hallucination-Gated All-Pass Rate above 0%, GPT-6 Luna (max), GPT-6.1 Sol (max), Muse Spark 1.3 (max), and Grok 4.7 (xhigh) set the Pareto frontier for score versus cost per task. GPT-6 Luna (max) is the cheapest at about $0.22 per task, scoring 3.3%, while Grok 4.7 (xhigh) leads at about $9.50 per task and Muse Spark 1.3 (max) costs about $4.20. The three Claude models cost about $18 to $22 per task.

AIOnce hallucinations are accounted for, Muse Spark 1.3 (max) drops from 26.7% to 8.9%, leaving Grok 4.7 (xhigh) first on the headline metric at 9.4%. Kimi K3 (max) falls from 16.7% to 5.3%, and Claude Sonnet 5.5 (max with fallback) falls from 11.7% to 2.8%. GPT-6.1 Sol (max) declines least, from 7.5% to 6.9%.

AIArtificial Analysis reports that Kimi K3 (max) achieves a 93.0% Criterion Pass Rate but averages 2.09 material hallucinations per task. Muse Spark 1.3 (max) scores higher at 96.0% while averaging 1.68 material hallucinations per task, showing that completing criteria and avoiding hallucinations are distinct skills.

AIArtificial Analysis compared six hallucination checkers on the same deliverables from 20 tasks across eight models. GPT-6 Sol and GPT-6 Luna generally flagged the most material hallucinations, while Claude Sonnet 5.5 and Gemini 3.8 Flash flagged far fewer, with Claude Opus 5.5 falling between Grok 4.7 and Sonnet. The counts reflect checker behavior rather than establishing accuracy or ruling out self-preference.

AIThe NVIDIA KGMON team placed second in the KDD Cup 2026 Data Agents competition with a system built around a smaller, clearer, and easier-to-verify agent harness. The competition required agents to answer natural-language questions over heterogeneous sources, including databases, CSV and JSON files, prose documents, PDFs, and briefing videos.

AITessl argues AI code review is slow because generated code outpaces trust, and proposes executable specs that let agents check preview environments against product intent. Its spec reviewer splits work between a planner agent that extracts requirements and parallel verifier agents that test each one against the code and base branch.

AITypeSafe AI's jev model was reported to be more calibrated than 40 other decision models tested on email classification. The main post itself only urges readers to stay calibrated, so no further figures or methodology are given.