Weight Initialization

Weight initialization is the strategy for assigning starting values to a neural network’s parameters before training begins, and the choice matters more than it might seem: a poor initialization scheme can cause a network to fail to train at all, while a well-chosen one enables stable, efficient convergence from the very first training step.

Why It Matters

If all weights start at exactly zero, every neuron in a layer receives an identical gradient during backpropagation and updates identically forever, meaning the network can never learn different features across neurons — a failure mode called symmetry that any working initialization scheme must break, typically by using small random values rather than a constant. Beyond avoiding this trivial failure, the scale of the random initial values also matters enormously in deep networks: values that are too large cause activations to grow explosively layer after layer (exploding gradients), while values that are too small cause them to shrink toward zero (vanishing gradients), both of which stall learning in very deep networks.

Common Schemes

Xavier (Glorot) initialization scales random initial weights based on the number of input and output connections for a given layer, aiming to keep the variance of activations roughly constant as data flows through the network — a good fit for networks using sigmoid or tanh activations. He initialization is a related scheme tuned specifically for ReLU-family activations, which zero out negative inputs and therefore need a different scaling factor to maintain the same variance-preserving property.

Relevance to LLM Training

Modern transformer training uses carefully tuned initialization schemes (often variants of these classic approaches, sometimes combined with techniques that scale initialization based on network depth) as one of several ingredients — alongside layer normalization, gradient clipping, and learning rate warmup — that together make it possible to stably train networks with dozens or hundreds of layers.

Initialization in the Era of Very Deep Transformers

As transformer models have grown to have dozens or hundreds of layers, some architectures use depth-aware initialization schemes that scale a layer’s initial weight variance based on its position in the network, since a fixed initialization scheme calibrated for a shallow network can behave differently once compounded across many more layers than it was originally validated on. Combined with pre-norm layer placement and careful learning rate warmup, these depth-aware adjustments are part of a broader set of engineering practices that collectively make it possible to train transformers hundreds of layers deep without the instability that would result from naively applying classic initialization schemes designed decades earlier for much shallower networks.