---
format: "aidr-story-markdown/v1"
id: "693b3130e25cc8808f585a694f7201006d36e37cb71c7b8ae2099287d6befd80"
canonical_url: "https://aidr.today/693b3130?lang=en"
title: "tokenizers v1: encode, decode and scaling, measured"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-09-21T00:00:00.000Z"
category: "Infra"
topics: ["huggingface","tokenizers","inference","open-source","scaling"]
source_urls: ["https://huggingface.co/blog/tokenizers-v1"]
summary: "We’re on a journey to advance and democratize artificial intelligence through open source and open science."
---

# tokenizers v1: encode, decode and scaling, measured

> [Open the canonical story](<https://aidr.today/693b3130?lang=en>)

**Published:** 2026-09-21T00:00:00.000Z
**Category:** Infra
**Topics:** huggingface, tokenizers, inference, open\-source, scaling

## Summary

We’re on a journey to advance and democratize artificial intelligence through open source and open science\.

## Sources

- [Story source](<https://huggingface.co/blog/tokenizers-v1>)