Decoding Phase

What is the decoding phase?

After prefill builds the KV cache, the decoding phase generates output one token at a time: predict a token, append it, attend over everything so far, repeat. Each step requires streaming the model’s weights through the GPU — for a single request, that means reading tens of gigabytes of weights to produce one token.

Why decode is bandwidth-bound, not compute-bound

Generating one token uses a tiny fraction of the GPU’s math capability but the full cost of moving weights from memory. The arithmetic is idle; the memory bus is saturated. Two consequences follow directly. First, per-request speed is capped by memory bandwidth — an H100 generates roughly the same tokens/sec for one user whether the model is busy or idle. Second, batching is nearly free: serving 30 concurrent requests reuses the same weight reads, multiplying aggregate throughput ~30× with modest per-request slowdown. This is the entire economic basis of continuous batching in vLLM and friends, and the reason TPS must always be qualified as per-request or aggregate.

Speculative decoding — cheating the sequential bottleneck

A small draft model guesses several tokens ahead; the big model verifies them in one parallel pass. Accepted guesses cost one step instead of many. On predictable text (code, boilerplate) speedups of 2–3× are common; on high-entropy creative text, acceptance rates drop and gains shrink — a rare optimization whose value depends on what your users write.

What people get wrong

• Optimizing decode with more FLOPS. Buying compute for a bandwidth-bound workload wastes money; bandwidth and batching are the levers.
• Capping output length as an afterthought. Every unnecessary output token is a full decode step; verbose system prompts that trigger verbose answers cost real latency and dollars.
• Benchmarking decode at batch size 1 and extrapolating to production, where continuous batching changes everything.

Primary source: PagedAttention/vLLM paper (Kwon et al., 2023)

Why Batching Helps Decoding More Than Prefill

Because the decoding phase is memory-bandwidth-bound rather than compute-bound, a GPU during decoding often has spare compute capacity sitting idle while it waits on memory access — this is precisely the inefficiency that continuous batching exploits, since serving multiple requests’ decoding steps together lets the GPU perform useful compute for several requests while paying the memory-bandwidth cost of loading the model’s weights only once per batch step, rather than once per individual request. This is why serving throughput (aggregate tokens per second across all users) scales so favorably with batch size during decoding specifically, even though it does relatively little to speed up any single request’s individual token generation rate.