Prefill Phase

What is the prefill phase?

When an LLM receives a request, it doesn’t start generating immediately. First it runs the prefill phase: every prompt token is pushed through the model in one large parallel pass, computing attention keys and values for the whole input and storing them in the KV cache. Only when prefill completes does the first output token appear — which is why prefill time largely is your TTFT.

Why prefill behaves differently from generation

Prefill is compute-bound: thousands of tokens processed simultaneously saturate the GPU’s matrix units. The decoding phase that follows is the opposite — memory-bandwidth-bound, one token at a time. This asymmetry drives real engineering decisions: a GPU that’s fast at prefill isn’t automatically fast at decode, and serving frameworks schedule the two differently (vLLM’s chunked prefill splits long prompts into pieces so ongoing decodes aren’t stalled behind a 100k-token newcomer).

The numbers that matter

Prefill cost scales roughly linearly with prompt length for compute, and attention scales quadratically in theory — in practice, a 100k-token prompt on a modern GPU takes seconds of pure prefill. That is why prompt caching exists: a cached prefix skips its share of prefill entirely, which is where the “up to 90% off and much faster TTFT” numbers come from.

What people get wrong

• Blaming the model for slow first tokens when the real cause is a bloated prompt: halving input length roughly halves prefill time — the cheapest TTFT optimization there is.
• Ignoring queueing: on busy servers, requests wait before prefill even starts; measured TTFT includes that wait, so p99 TTFT is often a capacity problem, not a model problem.
• Assuming prefill parallelism is free: one giant prefill can monopolize the GPU and freeze every in-flight generation — the exact problem chunked prefill was invented to fix.

Primary source: vLLM documentation — chunked prefill