SGLang Cuts 1T Model Restarts to 32 Seconds, Down From 8.8 Minutes

AntLingAGI and Alibaba developed a new memory optimization tool to accelerate the deployment of massive language models. The Weight Cache Daemon for SGLang reduced weight loading for the Ling-2.6-1T FP8 model to 0.63 seconds, which is up to 780 times faster than loading from disk. This optimization cuts total engine startup to 0.53 minutes, enabling the restart of a trillion parameter model in 32 seconds.

The tool targets the bottleneck of weight loading in production environments to improve recovery times and resource efficiency. SGLang serves as the serving framework for the deployment, and the daemon allows operators to bypass slow disk-based loading by caching model weights in memory.