I digged into Baseten's 2000TPS inference engine, here is what i found.
Baseten posted that it had built an inference engine for DeepSeek-V4.1-Flash in Rust and CUDA, reaching around 2,060 tokens per second on eight B300 GPUs. It also reported prefilling a million tokens in roughly twelve seconds.
I wanted to understand how this was possible. Getting even 500 TPS is hard. I was curious to learn about the design choices and optimizations that lead to such insanely fast inference. Lucky for us, they open sourced the code.
The DeepSeek-V4.1-Flash architecture
DeepSeek-V4.1-Flash is a causal encoder-decoder model. Its first 20 layers form the encoder, and the next 20 form the decoder. The decoder uses context supplied by the encoder, while a candidate pool feeds later sparse-indexing layers. The architecture also includes mixture-of-experts (MoE) layers and DSpark, a draft model used for speculative decoding.
The diagram from the DeepSeek-V4.1-Flash paper shows how these components fit together:
The inference configuration gives the dimensions of these components:
The model descripiton also identifies shared global KV, sparse indexing, bounded sliding-window replay and Engram conditional lookup memory as architectural features. These affect how much work the engine needs to do, particularly when processing a long prompt.
The model stores hundreds of billions of expert weights. How many does it use for one token?
Consider just the routed feed-forward experts. With hidden dimension d=5120 and intermediate dimension h=2304, one expert's three matrices (gate projection W_1, up projection W_3, and down projection W_2) contain
P_e = 3dh = 3 x 5120 x 2304 = 35,389,440
weights. The three matrices supply the two SwiGLU input projections and the output projection.
Across forty layers and 384 routed experts per layer, that becomes
40 x 6 x P_e = 8,493,465,600
Those 8.49 billion weights cover only the routed experts. The model also uses shared experts, attention projections, output heads and other components. Engram adds a large lookup table, but each token accesses only selected entries.
Baseten’s README reports both the output rate and the average time for one step of the engine’s generation loop:
| Configuration in the README | Decode rate | Average step time |
|---|---|---|
| 4 × GB300, TP4 | 1,740 tokens/s | 3.30 ms |
| 8 × B300, TP8 | 2,060 tokens/s | 2.71 ms |
TP4 and TP8 mean that one model execution is distributed across four or eight GPUs using tensor parallelism. These are GPUs cooperating on one request, rather than four or eight independent copies each producing that rate.
The decode example generates HTML for a Tetris game, with thinking disabled in the reproduction request. The README's million-token prefill is a separate result. It does not establish that the Tetris decode rate was achieved with a million tokens already in context.
What does 2,060 tokens per second require?
Rest of the post continues here with a better view.





