Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

A study of byte-level Transformers without fixed tokenizers finds that, with token-superposition training and hash embeddings, they outperform subword Transformers as model size scales. The models build local text abstractions using segmentation-like positions, and restricting up to 25% of intermediate layers to these local representations preserves downstream performance. Exploiting the learned structures for speculative decoding yields 3.4 times more accepted tokens than in subword Transformers.