---
format: "aidr-story-markdown/v1"
id: "13dc8541f1107ec7d39bb160ac38ce4fd0f6bef551e11d6cddef77fc0d388262"
canonical_url: "https://aidr.today/13dc8541?lang=en"
title: "Together AI Replaces Hand-Written Megakernels With AI Agents to Boost Inference Throughput 2.6x"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-11-17T20:07:09.000Z"
category: "Infra"
topics: ["meta","inference"]
source_urls: ["https://huggingnews.com/ai/together-ai-replaces-hand-written-megakernels-with-ai-agents-to-boost-in-68fff275","https://marketbrief.now/ai/together-ai-replaces-hand-written-megakernels-with-ai-agents-to-boost-in-68fff275"]
summary: "Together AI has transitioned from using hand-written megakernels to an agent-driven system called Together Compiler for optimizing AI inference. The new system uses AI agents to rewrite and fuse GPU kernels based on the specific model, chip, and traffic, which Together AI says can increase throughput by up to 2.6x. In tests on B200 GPUs, the company reported that Llama 3.2 1B reached 2,118 tokens/s, a 28% improvement over the performance of its previous manual megakernel approach. The company previously used megakernels to consolidate the entire forward pass of a model into a single persistent kernel, but a recent technical retrospective found this approach overcomplicated development and did not always maximize hardware limits. Together AI found that fusing smaller, distinct subsets of operations is more effective, though the number of possible combinations is too large for humans to optimize manually."
---

# Together AI Replaces Hand\-Written Megakernels With AI Agents to Boost Inference Throughput 2\.6x

> [Open the canonical story](<https://aidr.today/13dc8541?lang=en>)

**Published:** 2026-11-17T20:07:09.000Z
**Category:** Infra
**Topics:** meta, inference

## Summary

Together AI has transitioned from using hand\-written megakernels to an agent\-driven system called Together Compiler for optimizing AI inference\. The new system uses AI agents to rewrite and fuse GPU kernels based on the specific model, chip, and traffic, which Together AI says can increase throughput by up to 2\.6x\. In tests on B200 GPUs, the company reported that Llama 3\.2 1B reached 2,118 tokens/s, a 28% improvement over the performance of its previous manual megakernel approach\. The company previously used megakernels to consolidate the entire forward pass of a model into a single persistent kernel, but a recent technical retrospective found this approach overcomplicated development and did not always maximize hardware limits\. Together AI found that fusing smaller, distinct subsets of operations is more effective, though the number of possible combinations is too large for humans to optimize manually\.

## Sources

- [Story source](<https://huggingnews.com/ai/together-ai-replaces-hand-written-megakernels-with-ai-agents-to-boost-in-68fff275>)
- [Story source](<https://marketbrief.now/ai/together-ai-replaces-hand-written-megakernels-with-ai-agents-to-boost-in-68fff275>)

