Model Quantization & Perplexity Trade-offs

Introduction: The Squeezing of Parameters

Large Language Models (LLMs) are memory-heavy systems. A 70-billion parameter model stored at standard 16-bit floating-point (FP16 or BF16) precision requires 140 GB of VRAM just to load the weights.

Quantization is the process of compressing these weights to lower precision formats, such as 8-bit or 4-bit integers. This allows models to run on consumer hardware or fits larger, more capable models onto single enterprise GPUs. However, reducing precision discards information, which can degrade the model’s accuracy.

1. How Quantization Works: Linear Mapping

Linear quantization maps continuous floating-point numbers in a range $[X_{\min}, X_{\max}]$ to discrete integer slots in a target range (e.g. $[-128, 127]$ for 8-bit signed integers).

This mapping is defined by a scale factor $S$ and a zero-point offset $Z$ : $X_{\text{quantized}} = \text{round}\left(\frac{X}{S}\right) + Z$ $X_{\text{dequantized}} = S \times (X_{\text{quantized}} - Z)$

graph LR
    Float[Continuous FP16 Weight] --> Scale[Scale / Offset Normalization]
    Scale --> Int[Discrete 4/8-bit Integer]

Advanced Quantization Formats: NF4

In QLoRA, parameters are stored in NormalFloat 4 (NF4) format. NF4 is a data type optimized for weights that follow a normal (Gaussian) distribution. It divides the normal distribution curve into intervals containing equal numbers of parameters, minimizing quantization error compared to standard linear integer mapping.

2. The Perplexity Trade-Off

Perplexity is a mathematical metric that measures how well a model predicts a sample of text. A lower perplexity indicates the model is more confident and accurate in its predictions.

When a model is quantized, its perplexity increases. The severity of this degradation depends on the quantization level and model size:

Model Scale	Quantization Level	VRAM Size	Perplexity Change (WikiText-2)
Llama 3 8B	Unquantized (FP16)	16 GB	8.12 (Baseline)
Llama 3 8B	8-bit (INT8)	8.5 GB	+0.02
Llama 3 8B	4-bit (INT4)	4.8 GB	+0.15
Llama 3 8B	3-bit (INT3)	3.9 GB	+0.82 (Noticeable degradation)

The Scale Mitigation Principle

Larger models (such as 70B parameters) are more resilient to quantization degradation than smaller models (such as 8B parameters). A 70B model quantized to 4-bit precision often outperforms an 8B model at full 16-bit precision, making quantization a highly effective strategy for deploying larger architectures.

3. Real-world Deployment Implications

Throughput Benefits: Lower-precision weights reduce the amount of data that must be transferred from GPU memory to cache during processing. For memory-bandwidth bottlenecked tasks (like token generation), this can increase inference speeds.
Accuracy Trade-offs: While 8-bit quantization results in almost no accuracy loss, 4-bit quantization can cause minor degradation in reasoning tasks, and 3-bit or lower often leads to model incoherence.