Activation Checkpointing

Technical Overview of Activation Checkpointing

Activation Checkpointing (also known as Gradient Checkpointing) is a memory optimization technique that reduces the GPU memory footprint during neural network training.

During the forward pass of standard training, the GPU must cache intermediate layer activations to compute gradients during the backward pass. For deep models, storing these activations consumes significant VRAM. Activation Checkpointing addresses this by saving only a subset of activations (checkpoints) and recalculating the discarded activations on-demand during the backward pass.

Key Architecture & Implementation

The algorithm trades computational overhead for memory savings:

1. Forward Pass:
   • The model processes layers, but only saves activation tensors at designated checkpoint layers (e.g., at the boundary of every 4th transformer block).
   • Intermediate activations between checkpoints are discarded.
2. Backward Pass:
   • When backpropagation reaches a segment with missing activations, the model runs a local forward pass starting from the nearest upstream checkpoint to recalculate the needed tensors.
   • Once gradients are calculated, the recalculated activations are discarded again.

Standard:     F_L1 -> F_L2 -> F_L3 -> F_L4 (All activations saved in VRAM)
Checkpoint:   F_L1 (Save) -> F_L2 (Discard) -> F_L3 (Discard) -> F_L4 (Discard)
              Backward: Recompute L2 & L3 using saved L1 activation

Core Parameters

• Memory Savings: Can reduce activation memory footprints by up to 60-70%.
• Compute Penalty: Increases the execution time of the forward pass by approximately 30% due to the recalculation steps.

Real-world Applications

• Essential for training models with long sequence lengths or large batch sizes.
• Supported natively in PyTorch and distributed training libraries like DeepSpeed.