KV Cache Quantization

What is KV cache quantization?

At long context lengths, the KV cache — not the model weights — dominates GPU memory. A 70B-class model holds roughly 300+ KB of cache per token per sequence at FP16; a single 100k-token conversation consumes tens of gigabytes before a second user arrives. KV cache quantization stores those cached keys and values in FP8, INT8, or INT4, cutting that footprint 2–4× and directly multiplying how many concurrent long-context sequences a GPU can serve.

Why it’s not the same as weight quantization

Weight quantization compresses a static artifact once, offline, with careful calibration. KV cache is generated dynamically per request, so quantization happens on the fly, and its errors compound differently: every future token attends over the quantized cache, so early-token errors propagate through the entire generation. That’s why quality degradation shows up on long-range recall tasks (needle-in-haystack retrieval, long-document QA) before it appears in short-chat benchmarks.

The asymmetry practitioners exploit

Keys are empirically more sensitive to quantization than values — key errors distort which tokens get attended to, while value errors only blur what is retrieved. Mixed schemes (8-bit keys, 4-bit values) capture most of the memory win at a fraction of the quality cost, and per-channel scaling for keys is standard in serious implementations.

What people get wrong

• Validating on short prompts. The failure mode is long-range; a 500-token eval will pass while 80k-token recall quietly breaks.
• Stacking compressions blindly. 4-bit weights + 4-bit KV + aggressive batch sizes each cost a little quality; nobody measured the combination.
• Ignoring it as an API user — it’s one reason long-context pricing and quality vary across providers making otherwise-identical claims.

Primary source: KVQuant: Towards 10 Million Context Length Inference (Hooper et al., 2024)

Which Precision Levels Are Common

KV cache quantization typically targets INT8 or FP8 as a middle ground, since more aggressive 4-bit quantization of the cache tends to noticeably degrade output quality more than quantizing model weights to the same bit-width does — the cache stores dynamically generated values unique to each specific conversation, unlike model weights, which are fixed and can be more carefully calibrated in advance. Serving frameworks that support KV cache quantization typically make it a configurable tradeoff, letting operators choose between maximum quality (full precision cache, fewer concurrent users) and maximum capacity (quantized cache, more concurrent users or longer contexts per GPU) based on their specific application’s tolerance for any resulting quality degradation.