#Eval/Benchmark

AIMistral Large 4 reverse-engineered an unknown binary end-to-end, concluding it was Cobalt Strike and extracting its IoCs and malware configuration. The model also wrote a report with a YARA rule for detecting future incidents. The task, which could take a day of analyst work, was completed in 12 minutes.

AIMistral Large 4 solved 18 of 19 challenges in a CTF speedrun, with tool calls and solve times drawn from actual runs. The post frames the model as efficient at reasoning over diverse complex challenges compared with other models.

AIMistral AI says its Mistral Large 4 is one of the world's strongest AI models for cybersecurity. The post gives no benchmark scores, comparisons, or availability details.

AIA METR researcher discovered a bug in the transcript viewer of Inspect, a popular evaluation framework, that could let an agent show a user a fabricated or edited version of its transcript.

AIPerplexity says its open-weights multimodal decision model pplx-decider-v1.1-27b is now available and scores highest on Hugging Face's new Decision Index 0.3 benchmark. The model costs $0.02 per million input tokens, half the price of v1.

AIIn a Microsoft Research Podcast episode, Jennifer Neville discusses her nonlinear route into computer science and her push for more practical evaluations of today's AI systems. The post offers little beyond this framing, so no specific models, benchmarks, or results are mentioned.

AIARC Prize reports that Grok 4.7's reasoning-token usage correlates with its ARC-AGI-2 public scores. The low setting averaged about 10k tokens per test-pair attempt and scored 25%, while medium through xhigh used 86k to 120k tokens and scored 57.5% to 60%. ARC Prize suggests the lower token usage may help explain the low setting's lower score.

AIOn ARC-AGI-2, Grok 4.7's best score is 61.4% at $2.01 per task, down from Grok 4.6's 67.1% at $0.76 per task. The post notes cost per task rose 166% despite identical input and output token pricing.

AIGrok 4.7 used more reasoning tokens on average than Grok 4.6 on ARC-AGI-2 semi-private tasks at medium, high, and xhigh reasoning levels, raising its cost per task. Per test-pair attempt, medium used 136% more tokens, high 125% more, and xhigh 173% more, while low used 27% fewer. A chart compares the two models at xhigh on the 20 public tasks where Grok 4.7 increased token use the most.

AISpaceXAI's Grok 4.7 scored 90.2% on ARC-AGI-1 at $0.64 per task, higher than Grok 4.6, according to ARC Prize. It reached 61.4% on ARC-AGI-2 at $2.01 per task and 1.8% on ARC-AGI-3 under the standard harness ($2.7k), or 10.0% with the provider adapter harness ($4.8k), both lower than Grok 4.6 on those two benchmarks.

AIGoogle's EmbeddingGemma 2 is a 740M-parameter embedding model that outperforms some models more than twice its size while using about 191MB to 567MB of active RAM. It offers an 8K context window, 4x larger than the first generation, and can process up to 5.5 minutes of audio, 29 images, or 58 video frames in one pass.

AIGoogle Gemma says EmbeddingGemma 2 delivers best-in-size-class performance on MTEB (Code) and MIEB (Lite). The post highlights local codebase indexing, semantic code search, and coding agent retrieval, with a 9.92-point code performance gain over EmbeddingGemma.

AIOn ARC-AGI-1, DeepSeek V4.1 Flash scored 88.5% at high reasoning versus 90.5% at low, with high using 35% more output tokens without consistently better answers. On ARC-AGI-2, the reported per-task cost of max reasoning ($0.129) appears slightly lower than high ($0.133), but after excluding incomplete tasks caused by API issues, max is about 4.5% more expensive per task.

AIDeepSeek V4.1 Flash reaches 72.9% on ARC-AGI-2 at $0.13 per task and 94.5% on ARC-AGI-1 at $0.07 per task, according to ARC Prize verification. Compared with V4 Flash's best scores, it gains 11.5 points on ARC-AGI-2 and 5.5 points on ARC-AGI-1, but costs about 250% more per task.

AISemiAnalysis rated FarmGPU as ClusterMAX Underperform after its Slurm layer failed to advertise GPU resources and Kubernetes exposed no RDMA devices for scale-out networking. The post credits FarmGPU's Grafana monitoring, provisioning notes, and trustworthy technical team, while noting the team may be stretched thin across small clusters.

AIHamel Husain argues that similar wording does not show whether an LLM answer works for a specific application, so teams should check concrete failure modes. He adds that similarity metrics can still help with retrieval evaluation and measuring output diversity.

AISimon Willison reports that Mistral can now generate his pelican SVG test, shared via a Markdown SVG renderer. The post links to a rendered result but gives no benchmark or scoring details. Background from Mistral's own announcement describes Mistral Large 4 as a 1T-parameter, natively multimodal model with 49B active parameters, available via API today and with open weights planned for end of October.

AIMistral announced Mistral Large 4, a natively multimodal model with 1T parameters and 49B active, now available via API. Open weights are scheduled for release at the end of October, with a countdown page on Hugging Face showing October 31, 2026.

AIPerplexity's Computer ran two agents playing StarCraft against each other in real time, with the game never paused while each agent thought. Blue, playing with 41 Dragoons, lost the final match 2-5 to Red, which used High Templar and Psionic Storm after Blue failed to scout Red's build. Each agent received only its own fog-of-war-limited game state, and video input was not provided.

AIDuke University researcher Alexis Fox has been announced as a speaker at the ARC Prize Research Summit 2026. Fox studies how AI can reason over longer horizons and is lead author of PRO-LONG, which uses programmatic memory to support long-horizon reasoning.

AIMistral says its model outperforms GLM 5.3 in human evaluations across multiple domains. The post gives no benchmark scores, model names, or evaluation details.

AIMistral Large 4 is a 1T-parameter natively multimodal model with 49B active parameters, which the Mistral account says leads open-weights models from the US or Europe on aggregated benchmarks. The post claims it beats closed frontier models on visual grounding and posts strong results across cybersecurity, legal, and agentic behavior. It is available via API now, with open weights due at the end of October.

AIOn human evaluation, the model outperforms GLM 5.3 on STEM, CAD, and finance tasks and performs on par on agentic coding. The post is part 5 of a thread, so the model's name and other details come from earlier posts not included here.

AIMistral says its ML4 model reaches state-of-the-art performance among open models across a wide range of capabilities, and outperforms the best models in visual grounding, legal, and spreadsheet manipulation. The post reports ML4 ranks among the best on the AA Cyber Index, scoring 82% on vulnerability reproduction and patching and 93% on Cybench. It argues that self-hosted, auditable open models are the best defense option for enterprises today, and that they do not refuse to help.