1

Janas: Run LLMs larger than RAM by streaming experts from SSD

github.com

Janas runs large LLMs by streaming experts from an SSD to exceed RAM limits, achieving 37 tokens per second from Qwen3-Next-80B on a 32 GB laptop.

Sign in to suggest

Key sources

  1. DISCUSSIONkhalidkhairnews.ycombinator.com
Markdown