DeepSeek has open sourced the software that turns its model into answers at scale. On October 20, 2026, DeepSeek published DeepInfer, its in-house inference framework, on GitHub under the Apache 2.0 license (DeepInfer examples on GitHub, GitHub, retrieved 2026-10-26). DeepSeek describes it as the high-performance framework that powers its large-scale production services, and the first release serves one model, DeepSeek-V4.1-Flash, in offline mode. TechTarget reported the release on the same day, noting that it runs on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. The release is narrow by design, and that narrowness is the story: this is production code, not a general purpose server.
Key Takeaways
DeepInfer is DeepSeek's in-house inference framework, open sourced on October 20, 2026 under the Apache 2.0 license for DeepSeek-V4.1-Flash.
This release provides offline inference only, on Nvidia SM100 GPUs or Huawei Ascend 950 NPUs, with Linux, Python 3.12, and CUDA 13 or CANN 9.2.
It bundles DeepSeek's own kernels, expert parallelism, DSpark speculative decoding, and structured outputs, and it was built primarily on vLLM.
It leaves out a KV cache system, prefill and decode disaggregation, and load balancing across replicas, so it is not yet a complete serving service.
Its value today is clearest for model hosts and teams studying production inference, not for most developers, who can keep using the DeepSeek API.
To see why this matters, start with the cost side of AI. Training a model gets the headlines, but serving it is the daily bill. Every prompt, every agent step, and every token generated runs through inference software. Inference is the work of running an already trained model to produce answers. The same model can be cheap or expensive to serve depending on that software, which is why labs usually keep it private. DeepSeek has now published a large part of its own answer to that problem.
How this article was checked. We read the DeepInfer examples on GitHub. We compared them with TechTarget, Teck Hustlers, and Developers Digest. We kept vendor figures separate from independent findings. Where sources disagreed, we said so. For example, the star count shows attention, not performance.
What Is DeepSeek DeepInfer?
DeepSeek DeepInfer is a high-performance inference framework for DeepSeek-V4.1-Flash. Its default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, downloaded to the local Hugging Face cache on first use, according to the repository examples (DeepInfer examples on GitHub, GitHub, retrieved 2026-10-26). The entry point is a batch style offline example, python examples/offline_inference.py, not an HTTP server. That single fact sets expectations: you feed it prompts in batches and study the output, rather than pointing a live API at it.
The hardware list is equally specific. DeepInfer supports Nvidia SM100, the datacenter Blackwell generation, and Huawei Ascend 950 NPUs. It requires Linux, GCC 13 or newer, Python 3.12, and CUDA 13 for Nvidia or CANN 9.2 for Ascend, according to independent summaries of the README (Developers Digest, retrieved 2026-10-26). Consumer cards and older datacenter generations such as Hopper are not listed. If your hardware is not on that list, the repository is still useful as reading material, but it is not a runtime you can deploy as written.
Aspect | DeepSeek DeepInfer |
|---|---|
Announced | October 20, 2026 |
License | Apache 2.0 |
Model supported | DeepSeek-V4.1-Flash only, default checkpoint DeepSeek-V4.1-Flash-1017 |
Mode in this release | Offline inference only |
Hardware | Nvidia SM100 GPUs, Huawei Ascend 950 NPUs |
Build requirements | Linux, GCC 13 or newer, Python 3.12, CUDA 13 or CANN 9.2 |
Origins | Built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM |
Not included | KV cache system, prefill and decode disaggregation, load balancing across replicas |
What Is Inside the Release?
The kernel layer is where DeepSeek's earlier open work shows up. The release lists FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels (Teck Hustlers, retrieved 2026-10-26). These are the low level routines that make attention, matrix multiplication, and expert routing fast on the supported chips. Several were already open sourced by DeepSeek before this release, so DeepInfer gathers them into one working stack rather than introducing every piece from scratch.
Parallelism is the second layer. DeepInfer supports attention data parallelism and balanced context parallelism, plus mixture-of-experts expert parallelism with load balancing, known as EPLB. In plain terms, it splits work across chips in several directions at once: across requests, across long contexts, and across the experts inside a mixture-of-experts model. That is how a large model can run across a rack of accelerators instead of waiting on one chip.
Speculative decoding is the technique behind the DSpark system. A smaller draft step proposes several tokens ahead, and a confidence estimate decides which proposals are worth verifying with the full model. SiliconANGLE, as reported by Teck Hustlers, describes a draft of five tokens at a time. DeepSeek estimates that this cuts the draft tokens checked per step by about 30%. It also estimates a 4% to 7% throughput gain at production load (Teck Hustlers, retrieved 2026-10-26). Treat those figures as DeepSeek's own estimates, not independent measurements: no outside benchmark of DeepInfer's gains has been published yet.
Two features will interest teams that evaluate agents. Batch-invariant inference keeps results identical whatever else is in the batch, and deterministic sampling with a fixed seed makes repeated runs reproducible. Both are marked Nvidia only in the summaries of the README. A second technique replays only a short final slice of each prompt through the decoder layers, which DeepSeek says nearly halves total prefill computation. Structured outputs are also included: generation can be constrained to a regex, a JSON schema, or an EBNF grammar, which matters for tools and APIs that must return parseable results.
What DeepInfer Leaves Out, and Why It Matters
KV cache is the store of earlier prompt and generation state that lets a serving system avoid recomputing what it has already processed. DeepInfer does not include a KV cache system. It also does not include prefill and decode disaggregation, which is the separation of prompt processing and token generation onto different machines, or load balancing across data-parallel replicas. The README is candid about all three. It lists them as required for a complete inference service. It also notes they are not included in this release. Most of the engine functionality for disaggregation is already implemented (Developers Digest, retrieved 2026-10-26).
That gap explains the deployment scale in DeepSeek's own material. Coverage of the release notes that the smallest deployment DeepSeek describes as efficient uses 64 GPUs for prefill and another 64 for decoding (Teck Hustlers, retrieved 2026-10-26). This is a toolkit for cloud providers, inference hosts, and large AI teams, not a download for a single workstation. If you want a general purpose server for many models today, the honest comparison is still vLLM or SGLang, the engines DeepInfer itself builds on.
How to Run DeepInfer
First, check your hardware. On Nvidia, nvidia-smi can show the compute capability, and SM100 means compute capability 10.0. Then follow the repository path: create an empty Python 3.12 environment, set DEEPINFER_ACCELERATOR to nvidia or ascend, run ./build.sh, and run the offline example with python examples/offline_inference.py. The examples also show parallelism flags, such as --ep 8 for eight-way expert parallelism, and combinations with --dp and --cp for data and context parallelism. Set DEEPINFER_HF_MODEL_DIR to point at a local checkpoint if you do not want the default model downloaded on first use.
DeepSeek also released a companion library, deepseek-recipe, which converts requests into token sequences using the same logic as its official API, according to SiliconANGLE as reported by Teck Hustlers (Teck Hustlers, retrieved 2026-10-26). Matching the official API's token handling makes offline results comparable with production behavior.
For example, a host that already serves V4.1-Flash can read the kernel code. It can borrow the scheduling ideas. It does not need to adopt the whole stack. As a result, the useful read is often smaller than the headline. The headline is a full engine. The practical gain may be one kernel or one scheduling trick.
What Independent Coverage Says
Independent coverage agrees on the scope and differs on the emphasis. TechTarget frames DeepInfer as a cost story: token consumption is rising fast, and faster inference means fewer GPUs per workload, according to Lian Jye Su, an analyst at Omdia. The same report quotes Jack Gold, president of J. Gold Associates, on the messaging value. DeepSeek wants to be perceived as the open source leader in this space. Running on Huawei hardware helps Huawei build a reputation as a reliable alternative to Nvidia. However, Su notes Huawei still lags Nvidia in some areas.
The timing drew attention too. TechTarget notes that the release came the same week the United States banned federal agencies from using DeepSeek products, a move China described as an attempt to politicize technology. Releasing infrastructure under a permissive license does not change that policy picture, but it does let hosting companies anywhere serve DeepSeek's open-weight models more efficiently. That distinction, between the model, the code, and the policy around both, is worth keeping clear.
Early reaction was loud but not yet a verdict. The repository passed 8,000 stars by the afternoon of October 20, according to The Next Web as reported by Teck Hustlers. In contrast, the vLLM project thanked DeepSeek on X for crediting the open source community. It said it would keep improving its own V4.1-Flash support with ideas from DeepInfer. Stars measure attention, not performance. The performance question stays open until independent teams run DeepInfer on their own hardware and publish results.
One vendor number deserves a careful read. A deployment Q and A posted with the code gives one cluster figure. It says one DeepSeek cluster serves 155 trillion tokens a day. It also gives a 96% input cache hit rate. The average time to first token is around 400 milliseconds. The rate is 230 tokens per second, rising to 300 for coding (Teck Hustlers, retrieved 2026-10-26). Those are DeepSeek's production figures for its own system, not DeepInfer benchmark results on hardware you can rent, and they describe a full production setup including the serving pieces this release omits.
Who Should Use DeepInfer, and Who Should Not
DeepInfer is the right read, and possibly the right runtime, in four situations:
You host DeepSeek-V4.1-Flash and want DeepSeek's own kernels and scheduling ideas for that model.
You run Nvidia SM100 or Huawei Ascend 950 hardware and want to study or benchmark a production inference stack.
You build or evaluate agents and need batch-invariant or deterministic runs for reproducible results. Our llm-d 1.0 article covers a different answer to the same scale problem: a distributed platform for production inference across many models.
You maintain an inference engine and want to see what DeepSeek changed after building on vLLM, so you can borrow the pieces that fit your stack.
Skip it for now if you need a live HTTP service, support for models other than V4.1-Flash, or hardware outside SM100 and Ascend 950. Skip it too if you only want cheap DeepSeek tokens: the API already runs this code, and nothing in this release changes its pricing. If evaluating agents is your goal rather than serving them, our Microsoft Scope article covers an open platform for grading agents on tasks you define.
Frequently Asked Questions
Is DeepInfer free? Yes, the code is open source under the Apache 2.0 license. Your costs are the hardware it runs on, which means datacenter Blackwell GPUs or Ascend 950 NPUs, plus the engineering time to build and run it.
Can I run DeepInfer on my own computer? Almost certainly not as a deployment. This release supports only Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, and DeepSeek's efficient deployment scale is measured in racks, not single machines. You can read the code on any machine.
Does DeepInfer replace vLLM or SGLang? No. DeepInfer was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM, and it serves one model in offline mode. vLLM and SGLang remain the general purpose servers for many models and live traffic.
What is missing from DeepInfer? A KV cache system, prefill and decode disaggregation, and load balancing across data-parallel replicas. Those are the pieces that turn a fast offline engine into a complete online serving service, and DeepSeek says most of the disaggregation engine work is already implemented.
Are DeepSeek's performance numbers independently verified? No. The throughput, token volume, and latency figures in coverage of this release are DeepSeek's own production figures or estimates. No independent benchmark of DeepInfer's gains had been published at the time of writing.
The Bottom Line
DeepInfer opens the part of the AI stack that companies usually guard most closely: the software that turns accelerators into answers at a price users will pay. It is narrow, with one model, two chip families, and key serving pieces held back, and its headline numbers are still DeepSeek's own. But it hands every host, rival, and researcher a detailed look at how DeepSeek keeps its serving costs low, and it does so under a permissive license. The next test is not another announcement. It is an independent run on someone else's rack. Start with the DeepInfer examples on GitHub and the TechTarget coverage, or read the Indonesian version of this article.