DeepSeek open-sources DeepInfer, its inference framework for DeepSeek-V4.1-Flash
AIDeepSeek has open-sourced DeepInfer under the Apache-2.0 license, the in-house framework it says runs DeepSeek-V4.1-Flash at scale with high throughput and low latency. DeepSeek says it is sharing its production stack to help lower compute costs and energy use. A repost of DeepSeek's announcement gives 325 tokens per second for a single chat and about 1.7M tokens per second across 64 Nvidia Blackwell GPUs. The repost cites no independent measurement. Meng Shao says the release covers offline inference, is built primarily on vLLM with code from SGLang and TensorRT-LLM, and supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Shao says batch-invariant deterministic output works only on NVIDIA. Rohan Paul says the release omits a KV cache system, prefill and decode disaggregation, and load balancing across replicas. vLLM says it will apply ideas from DeepInfer to improve DeepSeek V4.1 Flash inference, and SGLang says it plans to bring some of its large-batch and multi-GPU ideas into SGLang.
- sources
- 11
- participants
- 17
In the past 48 hours
AILatest follow-upPost from wh