bitsandbytes

bitsandbytes is an open-source Python library that provides efficient CUDA implementations of low-precision (8-bit and 4-bit) operations for training and running large language models, and it’s one of the most widely used tools for making quantization and memory-efficient fine-tuning accessible without writing custom low-level GPU code.

What It Provides

The library offers drop-in quantized versions of standard neural network components — most notably 8-bit and 4-bit linear layers that can replace a model’s standard weight matrices with minimal code changes — along with 8-bit versions of common optimizers like AdamW, which reduce the memory needed to store optimizer states (which for AdamW normally require roughly twice the memory of the model’s own parameters, since it tracks two moving averages per parameter).

Its Role in QLoRA

bitsandbytes is best known as the enabling technology behind QLoRA, the widely adopted fine-tuning technique that quantizes a pretrained model’s frozen base weights down to 4-bit precision (using bitsandbytes’ NF4 data type, specifically designed to better represent the normal-distribution-like pattern typical of trained neural network weights) while training small full-precision LoRA adapters on top. This combination is what made it practical to fine-tune models with tens of billions of parameters on a single consumer or prosumer GPU, rather than requiring a multi-GPU server.

Practical Relevance

bitsandbytes integrates directly with Hugging Face’s transformers and peft libraries, typically requiring just a configuration flag (like load_in_4bit=True) rather than manual implementation, which is a major reason it became the default tool for memory-efficient fine-tuning in the open-source LLM ecosystem.

Beyond QLoRA: General 8-bit and 4-bit Inference

Outside of fine-tuning, bitsandbytes is also commonly used simply to load and run an existing pretrained model at reduced memory footprint for inference — a load_in_8bit=True or load_in_4bit=True flag when loading a Hugging Face model lets users run models that wouldn’t otherwise fit in available GPU memory, at some cost to inference speed and a typically small accuracy tradeoff. This makes it one of the most accessible entry points for individuals or small teams experimenting with large open-weight models on consumer GPUs, well before fine-tuning enters the picture at all.

The library also supports 8-bit optimizer states independent of the model weights themselves, which can be applied even during full fine-tuning to reduce optimizer memory overhead without necessarily quantizing the model weights at all.