TPS (Tokens Per Second)
Tokens Per Second — LLM speed measured two ways: per-request streaming speed users see, and aggregate GPU throughput that sets your unit economics.
Last reviewed: July 25, 2026
What is TPS?
TPS (Tokens Per Second) is the standard speed metric for LLM inference — and the most misquoted, because it means two unrelated things. Per-request TPS is how fast one user’s response streams. Aggregate TPS is how many tokens a GPU (or cluster) produces across all concurrent requests. A serving stack tuned for one deliberately sacrifices the other.
The trade-off that defines serving
Because the decoding phase is memory-bandwidth-bound, batching many requests together reuses each weight-read across the whole batch: aggregate TPS climbs steeply with batch size while per-request TPS degrades gently. Serving economics live in that gap. A GPU doing 60 tok/s for one user might sustain 2,000+ aggregate tok/s at high concurrency — a 30×+ difference in revenue per GPU-hour at the same hardware cost.
How much per-request TPS is enough?
People read at roughly 5–15 tokens/sec, so 20–40 tok/s streaming feels smooth for chat — beyond that, humans can’t tell. Machine consumers (agents parsing output, pipelines) benefit from every additional token/sec. Set targets by consumer, not vanity: paying for 150 tok/s per request in a chat product is optimizing past human perception.
What people get wrong
- Quoting TPS without batch size. “Model X does 100 tok/s” is meaningless — at what concurrency, what prompt length, which quantization?
- Ignoring the TTFT/TPS split. TTFT is prefill; TPS is decode; they have different bottlenecks and different fixes. A benchmark reporting one number hides which phase is slow.
- Comparing across output lengths. Short responses amortize prefill badly; measured “TPS” of a 20-token answer says little about a 2,000-token one.
Primary source: vLLM — performance benchmarking
Why the Two Measures of TPS Diverge
Per-request streaming TPS (how fast tokens appear to a single user) and aggregate throughput TPS (total tokens generated across all concurrent requests on a GPU) can move in opposite directions depending on serving configuration: batching more concurrent requests together typically improves aggregate throughput, since it makes better use of available GPU compute, but it can slightly slow down each individual request’s streaming speed, since the GPU is now dividing its attention across more simultaneous generations. This tension is a core tradeoff in LLM serving infrastructure design — providers tune their batching strategies to balance keeping individual users’ perceived speed acceptable against maximizing the total number of tokens (and therefore revenue, for a paid API) a given GPU fleet can generate per second.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.