KV Cache Eviction
An optimization strategy that discards low-attention key-value states to stay within VRAM bounds.
KV Cache Eviction discards low-importance key-value pairs (using metrics like attention weight or age) to prevent out-of-memory errors during long context window generations.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.