KV Cache Quantization

What is KV cache quantization?

At long context lengths, the KV cache — not the model weights — dominates GPU memory. A 70B-class model holds roughly 300+ KB of cache per token per sequence at FP16; a single 100k-token conversation consumes tens of gigabytes before a second user arrives. KV cache quantization stores those cached keys and values in FP8, INT8, or INT4, cutting that footprint 2–4× and directly multiplying how many concurrent long-context sequences a GPU can serve.

Why it’s not the same as weight quantization

Weight quantization compresses a static artifact once, offline, with careful calibration. KV cache is generated dynamically per request, so quantization happens on the fly, and its errors compound differently: every future token attends over the quantized cache, so early-token errors propagate through the entire generation. That’s why quality degradation shows up on long-range recall tasks (needle-in-haystack retrieval, long-document QA) before it appears in short-chat benchmarks.

The asymmetry practitioners exploit

Keys are empirically more sensitive to quantization than values — key errors distort which tokens get attended to, while value errors only blur what is retrieved. Mixed schemes (8-bit keys, 4-bit values) capture most of the memory win at a fraction of the quality cost, and per-channel scaling for keys is standard in serious implementations.

What people get wrong

• Validating on short prompts. The failure mode is long-range; a 500-token eval will pass while 80k-token recall quietly breaks.
• Stacking compressions blindly. 4-bit weights + 4-bit KV + aggressive batch sizes each cost a little quality; nobody measured the combination.
• Ignoring it as an API user — it’s one reason long-context pricing and quality vary across providers making otherwise-identical claims.

Primary source: KVQuant: Towards 10 Million Context Length Inference (Hooper et al., 2024)