The Production Sizing Guide to Scaling vLLM GPU Clusters

Introduction

Deploying large language models (LLMs) at scale requires balancing hardware costs with latency targets. Naive model hosting frameworks allocate static memory allocations that lead to memory fragmentation. vLLM addresses this bottleneck by using PagedAttention to manage the Key-Value (KV) cache.

To deploy vLLM in production, systems engineers must accurately calculate memory demands across model weights, activations, and the dynamic KV cache. This guide walks through sizing calculations and parallelism configurations to optimize GPU utilization.

---

The GPU VRAM Breakdown

An LLM instance requires VRAM partitioned into three main pools:

+-------------------------------------------------------------+
|                     Total GPU VRAM                          |
+------------------------------+----------------+-------------+
| Model Weights (Static)       | KV Cache       | Activations |
| (Parameters * Precision)     | (PagedAttention)| (Dynamic)   |
+------------------------------+----------------+-------------+

1. Static Model Weights

Model weight memory is static. The allocation depends on parameter count and quantization precision:

$\text{Memory}_{\text{weights}} = P \times \frac{B}{8}$

Where:

• $P$ is the parameter count in billions.
• $B$ is the precision bit-width (e.g., 16-bit for FP16/BF16, 8-bit for FP8/INT8, 4-bit for AWQ/GPTQ).

For example, a Llama-3-70B model loaded in BF16 (16-bit) requires:

$70 \times 10^9 \times \text{2 bytes} = 140\text{ GB of static VRAM}$

2. KV Cache (PagedAttention)

The KV cache stores the self-attention keys and values of past tokens in a request. PagedAttention divides this cache into blocks of virtual memory to eliminate fragmentation.

The VRAM allocated for a single token’s KV cache is computed as:

$\text{Bytes per Token} = 2 \times L \times H \times D \times 2$

Where:

• $L$ is the number of transformer layers.
• $H$ is the number of Key-Value heads (which is smaller than query heads in Grouped-Query Attention).
• $D$ is the head dimension size (typically 128).
• The first multiplier $2$ accounts for the separate Key and Value states.
• The final multiplier $2$ represents the 2-byte precision (FP16/BF16).

---

Parallelism Strategies: Tensor vs. Pipeline

When a model’s memory requirements exceed the capacity of a single GPU (e.g., an 80GB H100), the model must be distributed across multiple GPUs.

graph TD
    subgraph Tensor Parallelism (TP)
        TP0[GPU 0: Split Weights Col] <--> |NVLink Exchange| TP1[GPU 1: Split Weights Row]
    end
    subgraph Pipeline Parallelism (PP)
        PP0[GPU Node A: Layers 1-40] ===> |InfiniBand Transfer| PP1[GPU Node B: Layers 41-80]
    end

Tensor Parallelism (TP)

• Mechanism: Splits weight matrices of attention layers and feed-forward networks (FFNs) column-wise and row-wise across GPUs within the same node.
• Latency Profile: Very low overhead. Requires high-bandwidth interconnects like NVIDIA NVLink (up to 900 GB/s) because GPUs must synchronize activations at every layer.
• Rule of Thumb: Keep TP within a single physical server (node). Do not scale TP across nodes over networks, as ethernet latency will bottleneck execution speed.

Pipeline Parallelism (PP)

• Mechanism: Distributes layers sequentially across GPUs or nodes (e.g., layers 1–40 on Node A, layers 41–80 on Node B).
• Latency Profile: Higher overhead. Introduces GPU idle bubbles while nodes wait for upstream activations. Can be mitigated using micro-batching.
• Rule of Thumb: Use PP to scale models across multiple nodes connected via InfiniBand or RoCE v2 networks.

---

Sizing Matrix for Production Configurations

Here are production-ready hardware presets for common open-weight foundation models using vLLM:

Model	Target Precision	GPU Type	TP/PP Configuration	Target KV Cache Allocation
Llama-3-8B	BF16 (16-bit)	1x L40S (48GB)	TP=1, PP=1	0.90
Llama-3-70B	BF16 (16-bit)	4x A100 (80GB)	TP=4, PP=1	0.85
Llama-3-70B	FP8 (8-bit)	2x H100 (80GB)	TP=2, PP=1	0.90
Mixtral-8x22B	BF16 (16-bit)	8x H100 (80GB)	TP=8, PP=1	0.80

---

Optimization Directives for MLOps

To fine-tune your deployment settings, use the following startup command flags:

1. --gpu-memory-utilization: Sets the fraction of GPU memory to reserve for the model weights and the PagedAttention KV cache. The default is 0.90. If you run out of activation memory during peak batch sizes, reduce this to 0.85.
2. --max-model-len: Restricts the maximum context length of input prompts. Slicing this limit frees up substantial memory for the PagedAttention block pool.
3. --enable-chunked-prefill: Chunks massive input prompts to prevent execution spikes, stabilizing latency profiles in multi-user workloads.