InfraNov 18, 2026 · 03:071 source
Together AI Replaces Hand-Written Megakernels With AI Agents to Boost Inference Throughput 2.6x
From huggingnews.com
Together AI has transitioned from using hand-written megakernels to an agent-driven system called Together Compiler for optimizing AI inference. The new system uses AI agents to rewrite and fuse GPU kernels based on the specific model, chip, and traffic, which Together AI says can increase throughput by up to 2.6x. In tests on B200 GPUs, the company reported that Llama 3.2 1B reached 2,118 tokens/s, a 28% improvement over the performance of its previous manual megakernel approach. The company previously used megakernels to consolidate the entire forward pass of a model into a single persistent kernel, but a recent technical retrospective found this approach overcomplicated development and did not always maximize hardware limits. Together AI found that fusing smaller, distinct subsets of operations is more effective, though the number of possible combinations is too large for humans to optimize manually.
Sources
Content history (2)
- 2026-11-18 · Summary · vi · Replaced stored textTogether AI đã thay thế các megakernel viết tay bằng các AI agent nhằm tăng throughput in…Together AI đã chuyển đổi từ việc dùng các megakernel viết tay sang một hệ thống vận hành…
- 2026-11-18 · Title · vi · Replaced stored textTogether AI thay thế megakernel viết tay bằng AI agent để tăng throughput inference lên 2…Together AI thay thế megakernel viết tay bằng các AI agent để tăng throughput inference l…
One ranked edition, every morning
The stories that matter today. Free, cancel anytime.
Telegram @aidr_today