Grouped-Query Attention (GQA)

Technical Overview of GQA

Grouped-Query Attention (GQA) is an attention mechanism that groups query heads together to share single Key-Value (KV) projection heads.

Autoregressive LLM generation requires caching attention Key-Value states (the KV cache) for previous tokens to avoid redundant calculations. As context lengths scale, storing these states consumes significant VRAM. GQA addresses this by allowing multiple query heads to share a single KV head, maintaining model reasoning capabilities while reducing the memory footprint of the KV cache.

Key Architecture & Implementation

GQA bridges the gap between Multi-Head Attention (MHA) and Multi-Query Attention (MQA):

• MHA: Every Query head has a dedicated Key and Value head ($H_Q = H_{KV}$). This is memory-intensive but offers maximum representational capacity.
• MQA: All Query heads share a single Key and Value head ($H_{KV} = 1$). This minimizes memory use but degrades model reasoning quality.
• GQA: Query heads are divided into groups, and each group shares a single Key and Value head ($1 < H_{KV} < H_Q$).

MHA:  (Q1 -> KV1), (Q2 -> KV2), (Q3 -> KV3), (Q4 -> KV4)
GQA:  (Q1, Q2 -> KV1), (Q3, Q4 -> KV2)
MQA:  (Q1, Q2, Q3, Q4 -> KV1)

Core Parameters & Sizing

• Group Size ($G$): The number of Query heads sharing a single KV head, defined as: G = rac{H_Q}{H_{KV}}
• KV Cache Reduction: Memory usage scales down by a factor equal to the group size $G$. For example, a group size of 8 reduces KV cache VRAM requirements by 8x.

Real-world Applications

• Used in modern open-weight LLMs like Llama 3, Mistral, and Gemma 2 to support long context windows (up to 128k tokens).

Why It Became the Default Choice

Grouped-Query Attention sits deliberately between two extremes: standard multi-head attention, where every query head has its own dedicated key-value pair (maximum quality, maximum KV cache memory), and Multi-Query Attention, where all query heads share a single key-value pair (minimum memory, some quality cost). GQA divides query heads into a small number of groups, with each group sharing one key-value pair — for example, Llama 2 70B uses 64 query heads divided into 8 groups, meaning 8 query heads share each key-value pair rather than all 64 sharing one. This recovers most of the quality that full multi-head attention would provide while still delivering a substantial KV cache memory reduction, which is why it became the default choice for most large open-weight models released since 2023, effectively superseding both the original multi-head design and pure MQA for new large-scale models where serving efficiency at scale is a primary design consideration.