AI;DR
AI;DR
← Back to live feed1 story

Sunday, Oct 25, 2026

1

SGLang and vLLM Match DeepSeek's DeepInfer Engine With Full Production Serving Stacks

marketbrief.nowFrameworks2d

Open source inference frameworks SGLang and vLLM integrated DeepSeek's kernels and optimizations to match or exceed the performance of DeepInfer on the 552B parameter DeepSeek-V4.1-Flash model. SGLang achieved up to 1.04 times DeepInfer's prefill throughput on four Nvidia GB300 or B300 GPUs and up to 7.4% higher decode throughput across 64 GB300 chips. vLLM matched DeepInfer's decode curve across batch sizes 8 to 256 using half the expert parallelism and reached 954,000 tokens per second per GPU on the SemiAnalysis AgentX benchmark. Both frameworks provide the production serving infrastructure that DeepInfer omitted when it was open sourced under an Apache 2.0 license on Oct. 20.

Content history (2)
  • 2026-10-24 · Summary · vi · Replaced stored text
    SGLang và vLLM đã cập bằng được hiệu năng của DeepInfer engine do DeepSeek phát triển, đồ…
    Các framework inference mã nguồn mở SGLang và vLLM đã tích hợp kernels và tối u hóa mới n…
  • 2026-10-24 · Title · vi · Replaced stored text
    SGLang và vLLM đạt tốc độ tương đương DeepInfer của DeepSeek với ngăn xếp phục vụ toàn di…
    SGLang và vLLM trùng khớp DeepInfer của DeepSeek với bộ serving sẵn sàng production
Sign in to suggest

Key sources

  1. SOURCE@vllm_project“vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64.”x.com
  2. SOURCE@vllm_project“Scaled out with P/D disaggregation and Wide-EP on a GB300 NVL72 rack, vLLM serves V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU with 196 tokens/s P90 per user, 7.9× our single-node result.”x.com
  3. SOURCE@sgl_project“SGLang now matches DeepInfer performance on BOTH prefill and decode: • Prefill: Up to 1.04× throughput on 4× GB300 and 4× B300 • Decode: Up to 7.4% throughput on 64× GB300”x.com
  4. SUPPORT@sgl_project“DeepInfer's open-source release leaves out three key serving components: a KV cache system, prefill/decode disaggregation, and load balancing.”x.com
  5. SUPPORT@natolambert“DeepSeek is serving 150T tokens a day on one cluster, more than the open model inference on OpenRouter, Fireworks, Together, and BaseTen combined.”x.com
  6. SUPPORT@semianalysis_“DeepSeek dropped DeepInfer a few days ago, but the code is missing key components that is crucial for inference serving, and building them adds too much assumptions.”x.com
  7. SUPPORT@swyx“Currently, the service throughput of a single cluster is 155 trillion tokens/day, and the overall revenue is $38 per card per hour (based on off-peak prices).”x.com
  8. SUPPORT@tim_dettmers“DeepSeeks inference system is on par with OpenAIs stack and better what Anthropic is running (at least for their user size).”x.com
Markdown