● FrameworksOct 29, 2026 · 22:218 sources
SGLang Nears 1,000 Tokens a Second per User on DeepSeek V4.1 Flash With Nvidia GB300
From huggingnews.com
AI;DR
Open source inference framework SGLang reached nearly 1,000 tokens per second per user running DeepSeek-V4.1-Flash on an Nvidia GB300 NVL72 system, according to SemiAnalysis. The performance gain stems from combining prefill and decode disaggregation with kernel fusion techniques, and SGLang achieved up to 1.04 times DeepInfer's prefill throughput on four GB300 or B300 GPUs and up to 7.4% higher decode throughput across 64 GB300 chips.
Read originalSign in to suggest
Sources
+ 6 more
- SUPPORT@sgl_project“We first integrated Mega kernels (MegaMoE, MegaAttention, Mega-mHC, MegaGate), SWA bounded replay and Engram optimizations into SGLang to match DeepInfer's performance.”x.com
- SUPPORT@vllm_project“vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64.”x.com
- SUPPORT@bookwormengr“They spread out weights (experts) across 64 GPUs for decode, this means very little amount of weight per GPU (552B parameter, they use DP 64 + EP 64 resulting that 45B parameters per GPU, most of which are 4 bit”x.com
- SUPPORT@swyx“Currently, the service throughput of a single cluster is 155 trillion tokens/day, and the overall revenue is $38 per card per hour (based on off-peak prices).”x.com
- SUPPORT@zephyr_z9“Every 2nd/3rd tier inference provider now has a model and a recipe that can generate 90%+ margins on their compute”x.com
- SUPPORT@kevinsxu“DeepSeek open sourced its internal inference framework, Apache 2.0, no strings attached.”x.com