Aliases: Tokenization
Tokens
The subword units LLMs read and write; the billing and capacity unit for context windows, API pricing, and rate limits.
What are tokens?
LLMs don’t read characters or words — they read tokens, subword chunks produced by a tokenizer (usually byte-pair encoding). In English, one token ≈ 4 characters or ~¾ of a word; 1,000 tokens ≈ 750 words. “Understanding” might be one token; “misunderstandings” might be three.
Why developers must care
- Billing — every API is priced per million input and output tokens, with output typically 3–5× dearer.
- Capacity — the context window is a token budget.
- Rate limits — providers throttle on tokens per minute, not requests.
- Latency — output tokens are generated one at a time, so long answers are slow answers; input is processed in parallel and is much faster.
Cost reality
Token asymmetries surprise teams: code and JSON tokenize inefficiently (every brace and indent counts); non-English languages can use 2–4× more tokens for the same content, silently multiplying cost and shrinking effective context for international users. Verbose system prompts replicated across millions of calls are a classic five-figure line item — measure with the provider’s tokenizer, don’t estimate.
What people get wrong
- Counting words instead of tokens when checking whether content fits.
- Assuming tokenizers match across providers. The same text has different token counts on different models; recount when switching.
- Ignoring output caps. Models have separate maximum-output limits (often 4k–64k) well below the context window.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.