TTFT (Time to First Token)

What is TTFT?

TTFT (Time to First Token) measures how long a user waits before an LLM starts responding. In a streaming UI it is the perceived-latency metric: once tokens flow, people read along contentedly, but the silent gap before the first word is where they judge your product as fast or slow.

What TTFT is made of

TTFT = queueing time + prefill time + network overhead. Each piece has a different owner: queueing is a capacity/provisioning problem, prefill scales with prompt length, and network is usually noise unless you’re cross-region. Diagnose before optimizing — a p99 TTFT spike at peak hours is almost never a model problem and almost always a queueing problem.

Realistic targets

Sub-500 ms feels instant; under 1 s feels responsive; past 3 s users start rage-refreshing. Reasoning models complicate this: thinking tokens delay the visible answer by design, so products show “thinking…” states precisely because raw TTFT would otherwise be unacceptable. Interactive chat should budget TTFT strictly; batch pipelines shouldn’t care about it at all — paying for low TTFT on a nightly job is money burned (see the ~50% batch discount you’re forfeiting).

The levers, in order of effectiveness

Prompt caching (a cached prefix skips its prefill — routinely cuts TTFT several-fold on long stable prompts), shorter prompts (linear payoff), capacity headroom (kills queueing), and chunked prefill on self-hosted stacks (stops long prompts from stalling everyone else).

What people get wrong

• Reporting average TTFT. Users experience the p95/p99; averages hide the peak-hour queueing that drives complaints.
• Conflating TTFT with total latency. A response can start in 300 ms and stream for 40 s; which number matters depends entirely on whether output feeds a human or a parser.
• Comparing providers with different prompt lengths. TTFT scales with input; benchmark with your prompts or the comparison is fiction.

Primary source: NVIDIA — LLM inference metrics