● FrameworksNov 2, 2026 · 01:066 sources
Nvidia Tops 1,000 Tokens a Second per User on 700B+ Models With Agent Swarm Code
From marketbrief.now
AI;DR
A new Nvidia submission to the SemiAnalysis AgentX benchmark uses agent swarm software to exceed 1,000 tokens per second per user on models with more than 700B parameters. Running on Nvidia's Rubin NVL72 system, the submission delivers 20.9 times the performance per total cost of ownership compared to a GB300 NVL72 rack at 325 tokens per second per user. Open source inference framework SGLang reached 943 tokens per second per user at a concurrency of 1 using MegaKernel on a GB300 NVL72 rack.
Read originalSign in to suggest
Sources
+ 3 more
- SUPPORT@semianalysis_“GB300 NVL72 @sgl_project can get close to 1000 tok/s/user on DeepSeek Flash v4.1 by aggressively fusing kernels together”x.com
- SUPPORT@sgl_project“DeepSeek V4.1 Flash on GB300 NVL72, with SGLang’s aggressive kernel fusion pushing performance further”x.com
- SUPPORT@semianalysis_“Congrats to the SGLang and RadixArk team for their amazing work on MegaKernel + PD, improving both high concurrency throughput and low-latency serving as well”x.com