GPU Cluster Interconnects & Collective Communication

Introduction: The Memory Bandwidth Wall

When scaling deep learning models (such as Llama 3 405B or Mixtral 8x22B) across multiple GPUs, compute capacity is rarely the primary bottleneck. Instead, performance is constrained by interconnect bandwidth—the speed at which GPUs transfer data to one another.

In Tensor Parallelism (TP), the layers of a single transformer matrix are split across multiple GPUs. Because every attention head and feed-forward operation requires synchronization, the GPUs must exchange activation values during both the forward and backward passes. Without high-speed, low-latency interconnects, GPUs sit idle waiting for data (a state known as communication overhead).

1. Intra-Node Interconnects: NVLink vs. PCIe

Inside a single physical server (node) containing multiple GPUs, two primary bus architectures connect the processors:

PCI Express (PCIe)

PCIe Gen 5: Provides up to 128 GB/s of bidirectional bandwidth.
The Bottleneck: Because PCIe traffic must route through the host CPU’s memory controller, it introduces significant latency. This makes standard PCIe links too slow to support real-time Tensor Parallelism across multiple GPUs.

NVIDIA NVLink & NVSwitch

NVLink v4 (H100): Provides up to 900 GB/s of bidirectional bandwidth per GPU—more than 7 times faster than PCIe Gen 5.
NVSwitch: A physical routing switch built onto the server board that connects all GPUs directly to one another. This enables full-bandwidth communication across all GPUs inside the node simultaneously.

graph LR
    subgraph Single Node GPU Topology (NVLink/NVSwitch)
        GPU0[GPU 0] <--> NVSwitch[NVSwitch Hub]
        GPU1[GPU 1] <--> NVSwitch
        GPU2[GPU 2] <--> NVSwitch
        GPU3[GPU 3] <--> NVSwitch
    end

2. Inter-Node Interconnects: InfiniBand vs. RoCE

When scaling beyond a single server (e.g. connecting hundreds of H100 nodes), data must travel over an external network. Standard TCP/IP networking introduces too much CPU overhead and latency. Instead, distributed AI training and inference networks use specialized technologies:

InfiniBand (IB)

InfiniBand NDR (400 Gbps): A high-throughput, low-latency network technology widely used in supercomputing clusters.
RDMA (Remote Direct Memory Access): Allows a GPU in one server to read or write directly to the memory of a GPU in another server, completely bypassing the operating system and CPU on both sides.

RoCE (RDMA over Converged Ethernet)

RoCE runs RDMA protocols over standard Ethernet networks, avoiding the need for dedicated InfiniBand switches. While more cost-effective, it requires configuring Ethernet switches for lossless packet transmission to prevent performance degradation under high loads.

3. Collective Communication Primitives

Distributed training and inference frameworks use standard mathematical communication patterns, known as primitives, to synchronize data.

graph TD
    subgraph All-Reduce Operation
        Rank0[GPU Rank 0] --> Scatter[Reduce-Scatter]
        Rank1[GPU Rank 1] --> Scatter
        Scatter --> AllGather[All-Gather]
        AllGather --> Out0[Synchronized GPU Rank 0]
        AllGather --> Out1[Synchronized GPU Rank 1]
    end

1. All-Reduce

Concept: Combines tensor values from all GPUs (typically using addition) and distributes the final sum back to all GPUs.
Formula: The total data transferred over the network for a ring-based All-Reduce operation on data size $M$ across $P$ GPUs is: $\text{Data Transferred} = 2 \times \left(\frac{P - 1}{P}\right) \times M$

2. All-Gather

Concept: Each GPU starts with a local slice of data and shares it with all other GPUs, resulting in every GPU holding the complete, gathered dataset.
Usage: Commonly used in ZeRO-stage data parallel training to aggregate model parameters after optimizer steps.