DeepInfer is the inference framework DeepSeek uses to run its own large-scale production services, now published on GitHub under the Apache License 2.0. Based on the repository README as checked on October 20, 2026, it supports a single model, DeepSeek-V4.1-Flash, on two hardware targets, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and the public release covers offline inference. The pieces a public API needs, such as a KV cache system and load balancing, are not included. In short, it is a specialized engine core for DeepSeek's model, not a general-purpose replacement for vLLM or SGLang.

This guide explains what an inference engine does, what DeepInfer ships, what it leaves out, how it compares with the two most widely used open-source serving engines, and what to check before trying it on your own GPUs. Everything here reflects the first public version of the repository, so support may expand in later updates.

What is DeepInfer, DeepSeek's inference engine?

The README describes DeepInfer as the high-performance inference framework that powers DeepSeek's large-scale production services, and says this release provides offline inference with the core features needed for serving at scale. It was not written from scratch: the acknowledgments say it was originally built primarily on vLLM, with additional inspiration from SGLang and NVIDIA's TensorRT-LLM.

There is some history here. In an earlier open-infra-index post, DeepSeek explained that its internal engine was a heavily customized early fork of vLLM, tightly coupled to internal cluster-management tools, and that it would contribute standalone features to existing projects instead of releasing the whole engine. DeepInfer now puts a large share of that engine in its own repository, while the infrastructure-bound parts remain outside the release.

What does an LLM serving engine actually do?

Model weights on their own are just large files. The inference engine, often called an LLM serving engine, is the software that loads those weights onto accelerators and turns incoming requests into generated tokens. It decides how requests are batched, how the attention key-value (KV) cache is stored, and how a large model is split across many GPUs.

• Prefill: processes the whole input prompt in one pass. It is compute-heavy.
• Decode: generates the answer one token at a time. It is usually limited by memory bandwidth.
• Batching and scheduling: groups many users' requests so the hardware stays busy.
• Parallelism: spreads the model across devices. Mixture-of-experts (MoE) models often use expert parallelism (EP), placing different experts on different GPUs.

The same model can deliver very different tokens per second, time to first token and users per GPU depending on the engine. For any company running open-weight models on its own hardware, the engine choice is effectively a GPU cost decision. If you track how open-model and tooling releases spread through AI channels, the AI Telegram trends page below collects those discussions.

Page Reference

Trendstheddari.com›TrendsTelegram · AILive posts from AI channels

Key features inside DeepInfer

Kernels and parallelism

DeepInfer bundles kernel libraries DeepSeek has published separately, including FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels. For parallelism it combines attention data parallelism with balanced context parallelism, and MoE expert parallelism with expert-parallel load balancing (EPLB). Think of it as a stack tuned for the communication and compute patterns of DeepSeek's MoE architecture.

Faster decoding with DSpark

The README's DSpark feature uses draft tokens to accelerate decoding and confidence estimates to decide when to run verification, which places it in the speculative-decoding family. It is switched on with environment variables, and the default maximum draft length is 5.

Reproducibility: batch invariance and deterministic sampling

Batch-invariant inference keeps a request's results identical no matter which other requests it is batched with. Deterministic sampling returns the same output for the same input tokens, sampling parameters and fixed seed. Both are NVIDIA-only per the README. They matter most for evaluation runs and reinforcement-learning rollouts, where you need to rerun a job and get the same answer.

Other features

• Asynchronous scheduling: overlaps CPU batch scheduling with device execution; on by default.
• SWA bounded replay: reduces decoder prefill work by replaying only a bounded suffix for sliding-window attention layers.
• Structured outputs: constrains generation to a regex, a JSON schema or an EBNF grammar.

What is missing: an engine, not a full serving stack

The README has a dedicated list of components required for a complete inference service that are not part of this release:

1. A KV cache system that stores and reuses computed prompt prefixes across requests.
2. VE/prefill/decode disaggregation, which runs the stages on separate machines. The README notes that most of the required engine functionality is already implemented.
3. Load balancing across data-parallel replicas, for both prefill and decode.

You get an efficient engine for one group of accelerators, but not the fleet layer that accepts traffic from many users, routes it across replicas and reuses cached prefixes. The getting-started section centers on an offline inference example, and an OpenAI-compatible online server is not described.

DeepInfer vs vLLM vs SGLang

The core difference is scope. vLLM and SGLang are general-purpose engines for many models and many accelerators, while DeepInfer is purpose-built for one model on the newest hardware. The table reflects each project's README as of October 20, 2026.

The DeepInfer README does not include a same-conditions comparison against vLLM or SGLang. Its charts show DeepInfer's own throughput-versus-latency curves, so there is no evidence yet for claiming one engine is faster than another.

The projects are also more related than competitive. DeepInfer started from vLLM, and vLLM already lists FlashMLA among its attention kernels, so DeepSeek optimizations have a path into general-purpose engines. Watch whether techniques published in DeepInfer get picked up upstream.

Running models on your own GPUs: what to check first

1. 1

   Confirm your hardware

   If you are not on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs, you are outside the supported list. Teams on H100-class or consumer GPUs should stay on their current engine.

2. 2

   Confirm your models

   If you serve anything besides DeepSeek-V4.1-Flash, a multi-model engine like vLLM or SGLang keeps operations simpler.

3. 3

   Separate offline from online use

   Evaluations, synthetic data generation and RL rollouts are good first tests. A customer-facing API still needs your own cache, routing and disaggregation layers.

4. 4

   Benchmark under your conditions

   Compare against your current engine with your real prompt lengths and concurrency. The repository's run_sweep.sh, used for the README charts, is a reasonable starting point.

5. 5

   Check output quality

   The eval README gives Pass@1 reference figures for the V4.1 model: AIME 2026 99.9%, GPQA Diamond 93.4% and HLE-Full 42.6%. It calls them indicative rather than exact and suggests using them as a sanity check that a deployment is configured correctly.

6. 6

   Plan for maintenance and review

   With a short commit history, watch the update cadence and issue response. If your organization vets third-party code, include the bundled kernels and the vendor directory in that review.

How to read the viral numbers

Right after the release, community threads circulated figures about the throughput and per-GPU revenue of DeepSeek's whole service. Those numbers are not in the repository README. What the repository does document is a set of performance charts with their measurement setup and the evaluation reference scores. The scores show that the engine reproduces the model's quality; they are not a speed comparison.

• Open source does not mean cheap to run: the code is free, but SM100-class hardware and an operations team are not.
• The engine release is separate from API pricing: it does not change DeepSeek's API prices or terms by itself.
• Reproducibility features have conditions: batch invariance and deterministic sampling are NVIDIA-only, and deterministic sampling requires turning off the DSpark-related setting.
• Engine and weights have separate licenses: the engine is Apache 2.0, while the checkpoint follows its own model card.

Coverage of technical launches varies widely in accuracy. Comparing how blogs describe the release on the blog trends page below can help you spot claims that do not appear in the primary source.

Page Reference

Trendstheddari.com›TrendsBlog PostsSummarized Naver blog posts

What changes for open-source LLM serving

Three things change with DeepInfer. First, the kernel, parallelism and scheduling combination DeepSeek runs in production is now readable code. Second, teams serving DeepSeek-V4.1-Flash at scale on SM100 or Ascend 950 have an option beyond vLLM and SGLang. Third, reproducibility features such as batch-invariant inference and deterministic sampling are available in open source. Because the cache, routing and disaggregation layers are missing and model and hardware support is narrow, the practical move for most teams is to benchmark it against their current setup and watch whether its techniques land in the engines they already use.