---
format: "aidr-story-markdown/v1"
id: "cd872c282baf5f96f803f5a50598a2c403d63d33f8a725f17553da832c98f5e3"
canonical_url: "https://aidr.today/cd872c28?lang=en"
title: "Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-10-10T22:25:14.000Z"
category: "Research"
topics: ["llm"]
source_urls: ["https://arxiv.org/html/2610.05978v1","https://lobste.rs/s/t2fxpt/byte_language_models_scaling_emergent"]
summary: "A study of byte-level Transformers without fixed tokenizers finds that, with token-superposition training and hash embeddings, they outperform subword Transformers as model size scales. The models build local text abstractions using segmentation-like positions, and restricting up to 25% of intermediate layers to these local representations preserves downstream performance. Exploiting the learned structures for speculative decoding yields 3.4 times more accepted tokens than in subword Transformers."
---

# Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

> [Open the canonical story](<https://aidr.today/cd872c28?lang=en>)

**Published:** 2026-10-10T22:25:14.000Z
**Category:** Research
**Topics:** llm

## Summary

A study of byte\-level Transformers without fixed tokenizers finds that, with token\-superposition training and hash embeddings, they outperform subword Transformers as model size scales\. The models build local text abstractions using segmentation\-like positions, and restricting up to 25% of intermediate layers to these local representations preserves downstream performance\. Exploiting the learned structures for speculative decoding yields 3\.4 times more accepted tokens than in subword Transformers\.

## Sources

- [Story source](<https://arxiv.org/html/2610.05978v1>)
- [Discussion](<https://lobste.rs/s/t2fxpt/byte_language_models_scaling_emergent>)