DeepInfer GitHub: What DeepSeek Open-Sourced, What’s Missing, and Who Can Actually Run It

On 20 October 2026, DeepSeek pushed the first public commit of DeepInfer, the inference framework that serves its own DeepSeek V4.1 Flash traffic. Within a day the repository passed 16,600 GitHub stars. The Hacker News thread climbed past 2,200 points. The top comment, with 364 replies, quotes one sentence from DeepSeek’s notes: a single cluster serves 155 trillion tokens a day and earns US$38 per GPU-hour at off-peak prices.

That headline is real, but it is not what most teams will actually get from the repo.

What did DeepSeek actually release on GitHub?

The README calls DeepInfer “the high-performance inference framework that powers DeepSeek’s large-scale production services.” It is careful about scope: this release “provides offline inference with the core features needed for serving at scale.” Offline means you feed it prompts from a script; there is no HTTP server.

The only supported model is DeepSeek-V4.1-Flash, with DeepSeek-V4.1-Flash-1017 on Hugging Face as the default checkpoint. DeepSeek says inference code for future DeepSeek models will continue to be open-sourced in DeepInfer.

The README lists eight features:

• Optimized kernels from DeepSeek’s own libraries (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect) plus custom TileLang kernels.
• Parallelism: attention data parallelism and balanced context parallelism, MoE expert parallelism with expert-parallel load balancing (EPLB).
• DSpark speculative decoding with confidence-scheduled verification.
• SWA bounded replay, which cuts prefill work by replaying only a short suffix.
• Asynchronous, zero-overhead batch scheduling (on by default).
• Batch-invariant inference and deterministic sampling (NVIDIA only).
• Structured outputs constrained by regex, JSON schema or EBNF grammar.

The acknowledgments matter for anyone comparing engines. DeepInfer “was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.”

DeepSeek also open-sourced deepseek-recipe (github.com/deepseek-ai/deepseek-recipe), an MIT-licensed Rust component. It converts Chat Completions, Responses and Messages API requests into DeepSeek token sequences. It then parses the model’s output back into each protocol’s streaming format.

Why are H100s not on the DeepInfer hardware list?

DeepInfer supports exactly two accelerator families: “NVIDIA SM100 GPUs” and “Huawei Ascend 950 NPUs.” SM100 is NVIDIA’s name for compute capability 10.0, which NVIDIA’s CUDA GPU table assigns to the B200 and GB200. The H100 and H200 are compute capability 9.0.

The newer B300 and GB300 are listed as 10.3 in the same table. DeepSeek’s README does not mention them, so support there is unconfirmed. One Hacker News commenter is already looking for people to get it running on older SM80 cards.

The benchmark figures in the README were measured on 4 SM100 GPUs for prefill and 64 for decode, each inside one scale-up domain. That is the scale the engine is tuned for.

What is missing before DeepInfer can serve traffic?

The README lists three components that a complete inference service needs and that this release does not include:

1. A KV cache system: the store of attention results that lets a model skip re-reading a prompt it has already seen.
2. Vision encoder, prefill and decode disaggregation, meaning prompt reading and token generation run on separate GPUs (“most required engine functionality is already implemented”).
3. Load balancing across data-parallel replicas for both prefill and decode.

DeepSeek’s deployment Q&A explains why. The KV cache management and load balancing components “are tightly coupled to our internal infrastructure,” so that code is not yet fully open-sourced. DeepSeek describes both components in its Zhihu post (in Chinese), which the Q&A builds on.

The Q&A also gives the sizing those missing pieces were built for. It is the most useful reality check in the repo:

• Smallest efficient unit: one prefill plus one decode instance is deployable. But decode is recommended at EP64 or above, meaning the model’s experts are spread over at least 64 GPUs. Per the Q&A, “the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode.” That is 128 accelerators before the first token: a study project for most teams, not a weekend deployment.
• Decode parallelism: EP64 to EP128 (“Wide-EP”), using the largest expert-parallel size the available GPUs allow.
• KV cache storage: 20 SSDs of 30.73 TB per storage server, at least 40 GB/s read and 5 GB/s write, and about 14.4 TB of logical cache per online GPU.
• Retention: roughly 100 PiB of SSD in DeepSeek’s 3FS file system. Agent traffic keeps cache for 3 days, chatbot traffic for 7.

A Hacker News reply summed up the gap bluntly: DeepSeek released enough to study the system, not enough to point at a dozen of your own GPUs and get a production service.

What do the 155 trillion tokens and US$38 figures mean?

The deployment Q&A states that “a single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing).” Two cautions before you reuse those figures.

First, DeepSeek does not say which accelerator the US$38 refers to, a point one Hacker News commenter raised, without answer. Second, “valley pricing” refers to off-peak rates. The number describes revenue per GPU-hour at an off-peak tariff, not cost and not margin.

If you are tracking DeepSeek’s prices themselves, our breakdown of the DeepSeek V4.1 Flash 1017 release and its API prices covers the current rate card. Our check on whether DeepSeek raised its V4-Pro prices covers the rest of the lineup.

The more transferable numbers are the engineering ones, all from the same document. Expert-parallel load balancing keeps the slowest GPU within 5% to 9% of the average (ratios of 1.045 to 1.090 across the prefill and decode settings DeepSeek measured). Overall load balancing reaches “roughly 80% of a perfectly balanced offline benchmark.”

Cutting the KV cache time-to-live to about 3 days costs chatbot traffic 0.5% to 1% of cache hit rate, per the Q&A. That is why chatbot traffic keeps 7 days.

How does DSpark verify fewer draft tokens?

Speculative decoding lets a small draft model guess 5 tokens per step, which the main model then verifies in one pass. Instead of verifying all 5, DSpark predicts each draft token’s acceptance probability with a “Confidence Head.” It spends verification compute only on the tokens with the highest expected return.

In DeepSeek’s EP64/DP64 benchmark, this cuts the number of verified draft tokens by about 30% (from 5.00 to between 3.2 and 4.4). The average accepted length barely moves (3.29 versus 3.20). The gain grows with load:

At DeepSeek’s production operating point, around 60 concurrent sequences per rank, that is +4% to +7% throughput. At 16 sequences per rank it is slightly negative (-1.5%). If your deployment runs at low concurrency, fixed-length verification is still the better choice.

In DeepInfer, DSpark is enabled with DEEPINFER_ENABLE_MTP_VARLEN=1. DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length (default 5).

Which DeepInfer techniques are worth copying?

1. SWA bounded replay

Sliding-window attention (SWA) only looks at a short window of recent tokens, so its cache is cheap to rebuild. When that cache is missing, DeepInfer recomputes only the last 128 tokens, per the deployment Q&A. The cost becomes “a constant, independent of context length and layer count.”

Rebuilding it exactly would mean running 2,560 tokens through 20 decoder layers. The 128-token replay is described in the Q&A as “nearly halving total prefill computation.”

2. A prefill scheduler driven by a cost model

In DeepSeek’s prefill design, chunk size is not fixed. Each step gets a time budget in nanoseconds. A fitted quadratic model predicts how long a chunk of q new tokens will take given c already-cached tokens, and the scheduler solves for the largest q that fits, capped at 32,768 tokens.

Requests are first bucketed by total context length into four tiers: under 16K, 16K to 64K, 64K to 256K, and 256K or more.

3. Reproducible outputs

Batch-invariant inference keeps results identical across different batch compositions (DEEPINFER_BATCH_INVARIANT=1). Deterministic sampling adds a fixed seq_seed and disables variable-length speculative verification. Both are NVIDIA-only for now. They matter for evaluation pipelines and regression tests, where a changed answer must mean a changed model, not a different batch.

4. Special-token injection protection in deepseek-recipe

Strings such as <｜User｜> that appear in user input, tool results or tool-call arguments are encoded as ordinary text tokens, so text cannot smuggle in control tokens. On output, the end-of-reasoning marker and tool-call tags are recognized by token ID. A model that merely spells them out in plain text cannot trigger a tool call. Teams running their own agent stacks on any model can borrow this pattern.

How does DeepInfer compare with vLLM and SGLang?

DeepInfer and vLLM do different jobs.

Sources: the DeepInfer README and the vLLM documentation. SGLang plays the same general-purpose role as vLLM and is credited, along with TensorRT-LLM, in DeepInfer’s acknowledgments.

In practice, DeepInfer is a reference implementation and a source of kernels and scheduling ideas. vLLM and SGLang remain the engines most teams will deploy. Our bet: the most useful DeepInfer techniques will show up in those two projects before most teams run DeepInfer itself.

Should your team spend time on DeepInfer?

On that last row: on 20 October 2026, the same day DeepInfer went public, the Federal Register published DHS notice 2026-21447. The order it announces excludes DeepSeek products and services from federal civilian procurements.

It also orders the removal, from civilian agency IT, of DeepSeek products and of “third-party products and services developed using DeepSeek implementations.” An Apache-2.0 license does not settle whether a service built on DeepInfer falls under that wording for a given contract.

How do you run the offline example?

If you do have supported hardware, the README path is short. Create an empty Python 3.12 environment, then:

# or: ascend
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

The first run downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 into the Hugging Face cache; set DEEPINFER_HF_MODEL_DIR to use a local copy. The examples README in the repository shows parallel variants such as --ep 8 or --ep 2 --dp 8. examples/benchmark.py and examples/sweep/run_sweep.sh reproduce the throughput-latency curves.

Keep DEEPINFER_ACCELERATOR exported in every shell that runs DeepInfer, including the evaluation scripts for AIME 2026, HLE and GPQA Diamond.

Is DeepInfer the same as DeepInfra?

No. DeepInfer is DeepSeek’s open-source inference engine at github.com/deepseek-ai/DeepInfer. DeepInfra is an unrelated company that hosts AI models behind an API.

What we could not verify

DeepSeek has not said which accelerator the US$38 per GPU-hour refers to; only the company can settle that. The README does not mention the B300 or GB300, so whether the SM100 build runs on them is open. One Hacker News commenter says the DHS order itself dates from July 2026; the Federal Register notice published on 20 October 2026 is the only public text we have seen.

DeepSeek says the deployment Q&A “will be updated continuously” and that inference code for future models will land in DeepInfer. No date is given.

Sources

• deepseek-ai/DeepInfer on GitHub (README, repository metadata, commit history)
• DeepSeek V4.1 Flash Inference Deployment Q&A
• DeepSeek V4.1 Flash inference system, Zhihu post (Chinese)
• DeepSeek-V4.1-Flash-1017 on Hugging Face
• NVIDIA CUDA GPUs and compute capability
• vLLM documentation
• Hacker News discussion of the DeepInfer release
• Federal Register, DHS notice 2026-21447