NormalFloat 4 (NF4)

NormalFloat 4 (NF4) is a 4-bit numeric data type introduced as part of the QLoRA fine-tuning method, designed specifically to represent neural network weights more accurately than a standard 4-bit integer format would, by exploiting the fact that trained model weights tend to follow a roughly normal (bell-curve) distribution rather than being spread evenly across their range.

Why a Specialized Format Helps

A standard integer quantization format spaces its representable values evenly across a fixed numeric range. But neural network weights, once trained, tend to cluster densely around zero with progressively fewer values further from the center — a shape well approximated by a normal distribution. Using evenly spaced quantization levels on this kind of distribution wastes precision on rarely occurring extreme values while under-representing the densely populated region near zero, where most of the actual weight values live and where precision matters most for preserving model accuracy.

How NF4 Is Constructed

NF4 instead places its 16 representable values (4 bits allows 2⁴ = 16 distinct levels) at positions chosen to be information-theoretically optimal for a standard normal distribution — meaning each of the 16 quantization “buckets” is expected to contain roughly the same number of actual weight values, rather than the same span of numeric range. This concentrates NF4’s limited representational precision where it does the most good, given the known statistical shape of trained weights.

Practical Impact

In the QLoRA paper’s experiments, NF4 quantization of frozen base model weights, combined with training LoRA adapters in full precision on top, achieved fine-tuning results statistically comparable to full 16-bit fine-tuning — demonstrating that this distribution-aware 4-bit format loses meaningfully less accuracy than naive uniform 4-bit quantization would. NF4 is implemented in the bitsandbytes library and remains the default quantization data type for most QLoRA-based fine-tuning workflows.

NF4 vs. Standard 4-bit Integer Quantization

The practical difference between NF4 and a naive uniform 4-bit integer quantization scheme becomes most apparent at the tails of a weight distribution: uniform quantization spaces its 16 representable values evenly across the full observed range, meaning rare, extreme weight values consume representational “slots” that could otherwise capture finer distinctions among the much more common near-zero values. NF4’s distribution-aware placement instead concentrates its precision where weights are actually concentrated, which is why empirical comparisons in the QLoRA paper and subsequent work have consistently found NF4 preserves more model quality at 4-bit precision than naive uniform integer quantization applied to the same weights.