Decoding Phase
The token-by-token generation stage of LLM inference — sequential, memory-bandwidth-bound, and the reason output tokens cost more than input.
What is the decoding phase?
After prefill builds the KV cache, the decoding phase generates output one token at a time: predict a token, append it, attend over everything so far, repeat. Each step requires streaming the model’s weights through the GPU — for a single request, that means reading tens of gigabytes of weights to produce one token.
Why decode is bandwidth-bound, not compute-bound
Generating one token uses a tiny fraction of the GPU’s math capability but the full cost of moving weights from memory. The arithmetic is idle; the memory bus is saturated. Two consequences follow directly. First, per-request speed is capped by memory bandwidth — an H100 generates roughly the same tokens/sec for one user whether the model is busy or idle. Second, batching is nearly free: serving 30 concurrent requests reuses the same weight reads, multiplying aggregate throughput ~30× with modest per-request slowdown. This is the entire economic basis of continuous batching in vLLM and friends, and the reason TPS must always be qualified as per-request or aggregate.
Speculative decoding — cheating the sequential bottleneck
A small draft model guesses several tokens ahead; the big model verifies them in one parallel pass. Accepted guesses cost one step instead of many. On predictable text (code, boilerplate) speedups of 2–3× are common; on high-entropy creative text, acceptance rates drop and gains shrink — a rare optimization whose value depends on what your users write.
What people get wrong
- Optimizing decode with more FLOPS. Buying compute for a bandwidth-bound workload wastes money; bandwidth and batching are the levers.
- Capping output length as an afterthought. Every unnecessary output token is a full decode step; verbose system prompts that trigger verbose answers cost real latency and dollars.
- Benchmarking decode at batch size 1 and extrapolating to production, where continuous batching changes everything.
Primary source: PagedAttention/vLLM paper (Kwon et al., 2023)
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.