← Back to live feed1 story
NVIDIA engineers developed a decode-specialized kernel in Cute DSL that increased the inference throughput of the 2.8T parameter Kimi K3 model by 1.68x, reaching a peak speed of 423 tokens per second per user. The optimization, implemented as a 12,000 line of code Megakernel and enabled through aggressive Megakernel fusion as an upstream update to vLLM by the CentML team, surpasses the previous leaderboard high of 255 tokens per second. NVIDIA is currently extending these optimizations to other models, including DeepSeekv4.1.
Sign in to suggest
Key sources
- SOURCE@semianalysis_“NVIDIA has just achieved 423 tok/s/user 🚀 on the 2.8T Parameter Kimi K3 model via a new upstream vLLM optimization”x.com
- SUPPORT@artificialanlys“sets the fastest Kimi K3 result on our leaderboard at 255 tok/s per user”x.com
- SOURCE@zwljohnny“speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x)”x.com
- SUPPORT@tri_dao“A lot of the big wins weren’t new architectures, just people taking memory IO seriously.”x.com
- SUPPORT@_albertgu“there are still many ideas and optimizations waiting to be discovered for alternative architectures!”x.com
- SOURCEmarketbrief.now
- SOURCE@vllm_project“Kimi K3 decode on NVIDIA GB300 NVL72: 842 tokens/s at batch 1 with DSpark”x.com
- SUPPORT@vllm_project“reducing recurrent-state memory traffic by caching SSM inputs and updating the checkpoint only when needed”x.com