TPS (Tokens Per Second)

What is TPS?

TPS (Tokens Per Second) is the standard speed metric for LLM inference — and the most misquoted, because it means two unrelated things. Per-request TPS is how fast one user’s response streams. Aggregate TPS is how many tokens a GPU (or cluster) produces across all concurrent requests. A serving stack tuned for one deliberately sacrifices the other.

The trade-off that defines serving

Because the decoding phase is memory-bandwidth-bound, batching many requests together reuses each weight-read across the whole batch: aggregate TPS climbs steeply with batch size while per-request TPS degrades gently. Serving economics live in that gap. A GPU doing 60 tok/s for one user might sustain 2,000+ aggregate tok/s at high concurrency — a 30×+ difference in revenue per GPU-hour at the same hardware cost.

How much per-request TPS is enough?

People read at roughly 5–15 tokens/sec, so 20–40 tok/s streaming feels smooth for chat — beyond that, humans can’t tell. Machine consumers (agents parsing output, pipelines) benefit from every additional token/sec. Set targets by consumer, not vanity: paying for 150 tok/s per request in a chat product is optimizing past human perception.

What people get wrong

• Quoting TPS without batch size. “Model X does 100 tok/s” is meaningless — at what concurrency, what prompt length, which quantization?
• Ignoring the TTFT/TPS split. TTFT is prefill; TPS is decode; they have different bottlenecks and different fixes. A benchmark reporting one number hides which phase is slow.
• Comparing across output lengths. Short responses amortize prefill badly; measured “TPS” of a 20-token answer says little about a 2,000-token one.

Primary source: vLLM — performance benchmarking