DeepSeek has actually open-sourced the high-performance inference framework that powers its large-scale production services: DeepInfer
This is the first public release of the same technology stack DeepSeek runs live for its own services. This release focuses on offline inference, is licensed under Apache 2.0, and is mainly built on @vllm_project, with ideas borrowed from @sgl_project and TensorRT-LLM.
Open-source repository: https://github.com/deepseek-ai/DeepInfer
DeepSeek had previously open-sourced parts of its inference stack: FlashMLA (MLA attention kernel), DeepGEMM (matrix multiplication kernel), DeepEP (MoE communication library), EPLB (expert load balancing), and others, but it had lacked the "skeleton" that assembles them into a complete engine. DeepInfer fills that gap: it integrates these kernels and adds complete system-layer implementations for scheduling, parallelism, speculative decoding, and more. Together with the DeepSeek-V4.1-Flash model released at the same time, it forms a complete open-source inference solution of "model + engine." On the hardware side, it supports both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and dual-platform support is a distinctive feature.
Core Technical Breakdown
1. Kernels and runtime. Integrates FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom kernels written in TileLang (located in deepinfer/model_executor/layers/tilelang_ops). These are the low-level guarantees of throughput and latency.
2. Parallelism strategy: separate handling for attention and MoE. The attention part uses DP (data parallelism) or balanced CP (context parallelism, for handling very long sequences); the MoE part uses EP (expert parallelism) plus EPLB (expert-parallel load balancing), and supports redundant experts (32 by default, rounded up to be divisible by the EP count) to ease uneven expert load. This "attention DP/CP + MoE EP" combination is an architecture validated in DeepSeek's production.
3. DSpark speculative decoding. Accelerates decoding based on MTP (multi-token prediction) draft tokens. Its highlight is "confidence-based scheduled verification": the number of draft tokens accepted is decided by confidence rather than a fixed rule. The default maximum draft length is 5. This is one of the main sources of the inference speedup.
4. YoCo SWA bounded replay. An optimization for YoCo-style decoders with sliding window attention (SWA): during prefill, only a bounded suffix of tokens is retained and replayed (140 by default, which must cover the SWA window plus the maximum lookback length), greatly reducing the compute for decoder prefill. The repository also provides YoCo mask/compact equivalence verification.
5. Asynchronous scheduling. Batch scheduling on the CPU overlaps with execution on GPU/NPU, achieving a "zero-overhead scheduler," enabled by default.
6. Batch-invariant and deterministic inference. With DEEPINFER_BATCH_INVARIANT=1 enabled, results are fully identical across different batch compositions; combined with a fixed seq_seed and disabled MTP varlen, token-by-token deterministic sampled output can be achieved. This is critical for reproducible evaluation and regression testing (the repository uses per-sample exact-comparison PPL baselines as a check). Currently it is limited to the NVIDIA platform.
7. Structured output. Natively supports three types of constrained generation: regex, JSON schema, and EBNF, passed via so_args in SamplingParams.