Skip to content

Areas

Deployment & engineering Latest news

Engineering practice for running models: inference optimization, memory and cost, serving architecture, and infrastructure choices.

210 picksPast 30 days: 101 itemsTotal: 2,085 items

Latest pick

Top picks archive · Page 2

Oct 6

Oct 6TueItems 21–40
  1. vLLM BlogAI score62

    vLLM Speeds Up DeepSeek-V4.1-Flash Agentic Serving Through Kernel and Replay Optimizations

    Inferact and the vLLM community reported a 1.9× low-concurrency speedup and about 5.3× throughput under a 150 TPS constraint for DeepSeek-V4.1-Flash over three weeks. Gains came from SWA bounded replay with CUDA graphs, which cut TTFT by about 30%, and from integrated DeepSeek kernels such as MegaAttention, Mega-mHC, Mega-Gate, and DeepSelect. The post measures these results on the SemiAnalysis AgentX benchmark.

    AIWhy it matters: The post breaks down how SWA bounded replay and fused kernels cut prefill and decode costs, a reusable engineering pattern for long-context agentic serving.

  2. GitHubAI score72

    GitHub rebuilds Git infrastructure to handle agent-scale write volume

    GitHub reports that Git events on the platform rose from 218.2 billion to 473.3 billion per month between September 2025 and August 2026. It says agent workloads push write throughput and merge contention beyond what its current replica-based architecture handles well, so it is separating durable storage from compute while GitHub keeps running. The article states internal benchmarks reached up to 35 times higher write throughput.

    AIWhy it matters: The post links rising Git event volume to specific architectural bottlenecks, showing why agent workloads strain write paths and how GitHub plans to separate storage from compute.

  3. Mastra BlogAI score67

    Mastra launches Agent Controller GA, a runtime for long-running agent sessions

    Mastra has released Agent Controller in general availability, a runtime that hosts long-running agent sessions around the agent loop. The team says it was first built for Mastra Code and expanded to support Mastra Factory, which runs many concurrent sessions, and that memory usage in long-running Mastra Code processes dropped from 2–20 GB to 300–750 MB after optimizing UI state snapshots.

    AIWhy it matters: The post explains how the controller evolved from one developer's session to many concurrent sessions, with measured memory and storage changes useful to engineers building multi-user agent apps.

  4. Claude BlogAI score62

    Claude now works inside Google Docs, Sheets, and Slides in public beta

    Claude for Google Workspace is in public beta on all paid Claude plans, adding a sidebar to Google Docs, Sheets, and Slides. It can read the open file, edit text, build formulas, pivot tables, charts, and slides, and it asks for approval before changes unless the user chooses "Accept all edits." New Docs, Sheets, and Slides connectors in beta let Claude create and edit Google files from the chat, with access matching existing Google sharing permissions.

    AIWhy it matters: The source specifies how Claude edits Docs, Sheets, and Slides in place and where users keep control, which clarifies the practical workflow change.

  5. Claude BlogAI score62

    Comcast and Booz Allen use Claude Mythos to find exploit chains in codebases

    Comcast and Booz Allen used Claude Mythos Preview to find vulnerabilities that arise from interactions across code, configuration, and deployment rather than single-file bugs. Comcast identified a critical authentication flaw across 258 systems and about 170 million lines of code before any exploitation was observed. Booz Allen reported that one analyst reviewed eight production systems across 138 repositories in twelve days, a review its team estimated would have taken several months without the model.

    AIWhy it matters: The case studies show how security teams validate and remediate model-found exploit chains, a workflow relevant to anyone managing large codebases.

  6. Anthropic NewsroomAI score75

    Anthropic expands Cyber Verification Program into three tiered access levels

    Anthropic is launching an expanded Cyber Verification Program with three access tiers for qualifying security professionals, giving each tier different cyber capabilities and reduced blocking classifiers. On CyScenarioBench, Claude Opus 5.5 was blocked on 46 of 50 trials in the Defense Access tier, while the Red Team Access tier had no blocks and completed 34 of 50 tasks. Existing Project Glasswing members will move to the Specialized Access tier, and data retention is required for enrolled organizations.

    AIWhy it matters: The program lays out three verified access tiers with different cyber blocks, and its CyScenarioBench figures show how safeguards change what defenders can do.

Oct 5

Oct 5Mon
  1. Google Developers BlogAI score62

    EmbeddingGemma 2 releases multimodal embeddings with modular encoder loading

    Google released EmbeddingGemma 2, an open embedding model under the Apache 2.0 license that maps text, code, images, video, and audio into a shared 768-dimensional space. Developers can load a 270M-parameter text and code setup, or add vision and audio encoders up to a 740M-parameter full multimodal model. Matryoshka truncation to 256 or 128 dimensions reduces vector storage, with the guide noting quality losses on image, video, and speech retrieval at lower dimensions.

    AIWhy it matters: The guide gives concrete encoder sizes and dimension-storage tradeoffs, showing how to choose a configuration for text, code, image, video, and audio retrieval.

  2. GitHub Blog · AI & MLAI score63

    GitHub releases ReviewBench, an open benchmark for AI code review agents

    GitHub has released ReviewBench, an open benchmark for evaluating AI code review agents on 219 public pull requests across 19 languages. The benchmark reports grounded and augmented precision, recall, and F1 metrics, and its dataset, rubric, and judge are publicly available. GitHub says ReviewBench predicted the direction of a Copilot code review ensemble experiment's production results before A/B testing.

    AIWhy it matters: The post explains how ReviewBench was built and validated, and reports an offline-to-production comparison that shows how well a benchmark predicts real experiment outcomes.

  3. Liquid AI · new models on Hugging FaceAI score67

    Liquid AI releases d1-3B, a 3B multimodal decision model for edge deployment

    Liquid AI has released d1-3B, a 3B parameter multimodal model post-trained to return calibrated, typed answers to yes/no, choice, and score questions in one forward pass. The source reports a Decision Index 0.2.1 score of 48.57, the highest among models under 10B in its table, and 8 ms per decision on an NVIDIA RTX 4090.

    AIWhy it matters: The source gives benchmark scores against named peer models and edge latency figures across several hardware targets, helping readers judge fit for on-device decision pipelines.

Oct 4

Oct 4Sun
  1. Liquid AI BlogAI score70

    Liquid AI releases d1 decision model with image input support

    Liquid AI introduces d1, its first decision model, now accepting both text and images. The company says d1 matches or beats GPT-6.1 Sol on four of six tested applications, at 19x to 200x lower cost and with faster answers on every task. d1 is available on the Liquid AI API and through Vercel and OpenRouter, with text-only support on those two platforms for now.

    AIWhy it matters: The post gives benchmark comparisons against named models along with per-token pricing and latency figures, which makes the cost and speed tradeoff checkable.

  2. Epoch AIAI score62

    OpenAI researchers' coding-agent usage is doubling about monthly, Epoch AI reports

    OpenAI researchers' daily coding-agent usage, valued at API prices, rose from under $1 in January 2026 to $601 for the median researcher by mid-August. The 90th-percentile researcher reached over $7,000 per day, and both groups show doubling times of roughly one month. Epoch notes these are API-list values, not OpenAI's internal costs.

    AIWhy it matters: The figures show internal coding-agent usage growing fast enough to matter for research cost, though they measure API-list value rather than OpenAI's actual spending.

Oct 2

Oct 2Fri
  1. Baseten BlogAI score70

    Baseten's agent-built VibeQwen engine beats vLLM on Qwen-3.6 decode speed

    Baseten tested the MetaInfer skills-only approach by having Claude Code build an inference engine, VibeQwen, for Qwen-3.6-35B-A3B in NVFP4 on a single B200. On single-stream text, VibeQwen decoded 90% faster than a tuned vLLM 0.25.1 deployment (1,792 vs. 943 TPS) and cut time to first token from 28 ms to 12 ms, with a 71% throughput gain at concurrency 32. The author notes this was an outcome-focused run that allowed some numerically different outputs as long as accuracy stayed at or above the BF16 baseline.

    AIWhy it matters: The post tests a skills-only inference engine method on a real model and states the speed and accuracy constraints used, helping readers judge how far such automated optimization can be trusted.

  2. Epoch AI · The Epoch BriefAI score62

    Epoch AI estimates 2026 compute could run hundreds of millions of AI agents

    Epoch AI estimates that compute built from projected 2025 to 2027 high-bandwidth memory shipments could support tens to hundreds of millions of frontier AI agents, or billions of cheaper ones. Running nonstop, the top-tier agents would match the working hours of 140 million to 700 million full-time employees, and the central DeepSeek V4 Pro estimate of about 1.9 billion agents would match 8 billion workers.

    AIWhy it matters: The estimate converts memory shipments into agent capacity and revenue ranges, showing how hardware supply could translate into labor and sales if demand keeps up.

  3. Google ResearchAI score60

    Google's TEE-based federated learning system adds verifiable privacy guarantees

    Google announces a next-generation federated learning system that uses Trusted Execution Environments to provide verifiable, auditable data anonymization. The system publishes access policies to a public transparency log and is deployed in Gboard, which has launched English and Japanese next-word prediction models with stronger privacy guarantees and improved accuracy. Training time has also sped up significantly because computation moved to the server and is parallelized across many machines.

    AIWhy it matters: The post shows how Trusted Execution Environments make federated learning's privacy claims externally verifiable, rather than relying on trust in the server operator.

  4. Ai2 (Allen Institute for AI)AI score67

    Ai2 open-sources AstaBrief 8B, a fast open-weights scientific report model

    Ai2 released AstaBrief 8B, a model that turns a research question and retrieved literature excerpts into a cited report, along with its training data. In Asta's Generate a report feature, Fast mode averages 51.1 seconds per report versus 178.5 seconds for Thinking mode, about 3.5x faster. The model is built on Qwen3-8B with supervised fine-tuning and DPO, and institutions can run its open weights on their own infrastructure.

    AIWhy it matters: The post explains the data filtering and one-pass generation choices behind a fast open-weights report model, showing what worked and what did not.

  5. Hugging Face BlogAI score62

    AutoSynthData generates targeted training data for enterprise agents from failures

    ServiceNow CoreAI introduced AutoSynthData, which uses a target model's failures and a stronger teacher's successes to generate and validate new agent training tasks. In EnterpriseOps Gym experiments, the Hybrid domain produced 2,000 samples and raised Gemma-4-26B-A4B-it mean Pass@1 by 7.2 percentage points, while the ITSM domain produced 1,994 samples and raised it from 18.77% to 27.18%.

    AIWhy it matters: The post shows how failure analysis, teacher demonstrations, and verifier checks combine into a repeatable pipeline for generating targeted agent training data.

  6. Prime Intellect BlogAI score67

    Prime Inference launches serverless and reserved serving for open frontier models

    Prime Inference is a serving platform for frontier open-source models, offering serverless endpoints and reserved capacity on Prime's GPU infrastructure across multiple datacenters. Its first public deployment, GLM-5.3, went live on OpenRouter on September 22, and the post reports a near-zero tool-call error rate and 100% uptime since launch. The post also describes GLM-5.3 serving on GB200 NVL72 with prefill/decode disaggregation and NVFP4 KV compression.

    AIWhy it matters: The post separates scheduler, KV-cache, and tool-call fixes, showing concretely which bottlenecks shape production serving of open frontier models.

Oct 1

Oct 1Thu
  1. Epoch AIAI score62

    Epoch AI estimates how many concurrent AI agents 2025–27 memory shipments could run

    Epoch AI estimates that high-bandwidth memory shipped in 2025–27 could eventually support about 30–170 million concurrent frontier-model agents once fully deployed and allocated. Using DeepSeek V4 Pro serving benchmarks, the estimate rises to about 1.9 billion concurrent agents. The authors compare the implied API-equivalent spending of $2.6–5.3 trillion per year with projected developer revenue of roughly $1 trillion by end-2027, suggesting demand may lag supply.

    AIWhy it matters: The analysis converts HBM shipment data into concurrent agent capacity and compares it with projected API revenue, showing where compute buildout may outpace demand.

  2. NVIDIA BlogAI score62

    NVIDIA Blackwell GPUs power OpenAI's GPT-6 Astra Ultrafast mode in API

    GPT-6 Astra Ultrafast, running on NVIDIA Blackwell GPUs, is now available in the OpenAI API and to eligible ChatGPT Work and Codex users. The source says Ultrafast offers up to 8x faster token generation than Astra Standard mode, which can shorten coding agents' response times between tool calls. OpenAI also says it uses its own models to keep optimizing inference software on NVIDIA GPUs after deployment.

    AIWhy it matters: The source ties a specific speed claim to coding agents' edit-test-debug loops, showing where faster token generation changes developer workflows.

  3. Goodfire ResearchAI score60

    Goodfire proposes protein embedding monitors for biosecurity risks in AI agents

    Goodfire Research developed sequence-aware monitors using protein language model embeddings to flag concerning biological sequences in dual-use AI agent tasks. On a custom benchmark, the monitors outperformed frontier model safeguards with fewer refusals on benign requests, and they held up better against paraphrasing and fragmentation attacks. The paraphrase results rely on in-silico estimates and do not establish whether the redesigned proteins keep biological activity, and the monitors run in milliseconds per sequence.

    AIWhy it matters: The post gives a concrete benchmark setup and fragmentation results, showing how sequence embeddings can separate dual-use biology requests that task-based safeguards handle poorly.