Tokens

What are tokens?

LLMs don’t read characters or words — they read tokens, subword chunks produced by a tokenizer (usually byte-pair encoding). In English, one token ≈ 4 characters or ~¾ of a word; 1,000 tokens ≈ 750 words. “Understanding” might be one token; “misunderstandings” might be three.

Why developers must care

• Billing — every API is priced per million input and output tokens, with output typically 3–5× dearer.
• Capacity — the context window is a token budget.
• Rate limits — providers throttle on tokens per minute, not requests.
• Latency — output tokens are generated one at a time, so long answers are slow answers; input is processed in parallel and is much faster.

Cost reality

Token asymmetries surprise teams: code and JSON tokenize inefficiently (every brace and indent counts); non-English languages can use 2–4× more tokens for the same content, silently multiplying cost and shrinking effective context for international users. Verbose system prompts replicated across millions of calls are a classic five-figure line item — measure with the provider’s tokenizer, don’t estimate.

What people get wrong

• Counting words instead of tokens when checking whether content fits.
• Assuming tokenizers match across providers. The same text has different token counts on different models; recount when switching.
• Ignoring output caps. Models have separate maximum-output limits (often 4k–64k) well below the context window.

Why the Token/Word Distinction Matters

Because pricing, rate limits, and context windows are all measured in tokens rather than words or characters, understanding roughly how tokens map to text is practically useful: a common rule of thumb for English text is about 4 characters or roughly ¾ of a word per token, meaning 1,000 tokens is typically 700-750 words of English prose. This ratio varies considerably by language and content type — languages with less representation in a tokenizer’s training data (many non-Latin-script languages) or content with unusual formatting (dense code, or text full of rare technical terms) can require noticeably more tokens per unit of actual content, sometimes 2-3x more than equivalent English prose, which has real cost and context-window implications for multilingual or code-heavy applications. This is also why identical text can cost differently on different model providers, since each uses its own tokenizer trained on its own vocabulary — a prompt that consumes 500 tokens on one model’s tokenizer might consume a meaningfully different count on another’s.