What Is Model Quantization? The Plain Explanation, With Numbers

Model quantization stores a neural network's weights at lower numerical precision so it needs less memory and runs faster. This article explains the arithmetic behind it, the difference between post-training quantization and quantization-aware training, what GGUF, GPTQ and AWQ actually mean, and how to choose a bit width for your hardware without wrecking model quality.

The short answer

Model quantization is the process of storing a neural network's weights, and sometimes its activations, in a lower precision number format than the one it was trained in. A model trained in 32-bit floating point gets converted to 8-bit or 4-bit integers. Same architecture, same parameter count, smaller numbers holding each parameter.

The reason anyone bothers is memory. Llama 3 8B in 16-bit precision needs about 16 GB just to hold the weights, before you add the KV cache or the activations. The same model in a 4-bit GGUF build lands around 4.9 GB, which fits on a mid-range gaming GPU or an M-series MacBook with room left over. Nothing was removed. The numbers were just written down less precisely.

That is the whole idea. Everything else is detail about how to do it without the model getting noticeably dumber.

What each precision level actually costs you

Bits per weight translates directly into gigabytes. Multiply parameters by bytes per parameter and you have your memory floor. An 8 billion parameter model at one byte per weight is 8 GB, and there is no arguing with that number.

Here is the ladder people actually use in 2025.

Format	Bits per weight	8B model weights	Where it shows up
FP32	32	~32 GB	Training reference, older research code
FP16 / BF16	16	~16 GB	Standard training and server inference
FP8	8	~8 GB	Newer NVIDIA hardware, training and inference
INT8	8	~8 GB	Production server inference, mobile
INT4	4	~4.5 to 5 GB	Local LLMs, consumer GPUs, laptops
2-bit / ternary	2 or less	~2 GB	Research, aggressive edge deployment, usually lossy

How the conversion actually works

Take a tensor of weights. Find its minimum and maximum. Now map that whole range onto the 256 slots an 8-bit integer gives you, using a scale factor and an offset called the zero point. To use the weight during inference, you multiply the integer back by the scale. That is affine quantization, and it is genuinely that simple.

The trouble is what happens when one weight in the tensor is enormous. A single outlier stretches the range, and every other weight gets squeezed into a handful of integer slots. Precision collapses for the 99 percent because of the 1 percent.

So nobody quantizes a whole tensor with one scale anymore. You do it per output channel, or per group of 128 weights, each with its own scale factor. It costs a little extra storage for all those scales and it buys back most of the accuracy. This matters more for large language models than for small vision models, because researchers found in 2022 that transformers above roughly 6.7 billion parameters develop systematic outlier features in specific dimensions, and naive 8-bit quantization of those models falls apart in a way that a ResNet never did.

Activations are harder than weights. Weights sit still and you can inspect them offline. Activations change with every input, so you need calibration data, a few hundred representative samples, to estimate the ranges they will occupy at runtime. Many practical setups quantize the weights and leave the activations in 16-bit for exactly this reason.

Post-training vs quantization-aware, and the file formats you will meet

Post-training quantization takes a finished model and compresses it. Minutes to hours, no gradient updates, works with a small calibration set. This is what almost everyone does.

Quantization-aware training simulates the rounding error during training or fine-tuning, so the model learns weights that survive the conversion. It costs a real training run. You do it when post-training quantization has already failed and you still need that bit width, which usually means you are going below 4 bits or targeting a tiny embedded chip.

The names you will run into when you download a model are mostly post-training methods.

• GGUF: the llama.cpp file format. Variants like Q4_K_M and Q5_K_M mix bit widths across layers, keeping sensitive layers wider. Best choice for CPU and Apple Silicon.
• GPTQ: 4-bit weight quantization that adjusts remaining weights layer by layer to compensate for rounding error. GPU inference.
• AWQ: identifies the small fraction of weights that matter most for activations and protects them. Often slightly better quality than GPTQ at the same size.
• bitsandbytes: the load_in_4bit and load_in_8bit flags in Hugging Face. Zero effort, quantizes on load, slower than a purpose-built kernel.
• QLoRA: quantize the base model to 4-bit, freeze it, train small LoRA adapters on top. This is how people fine-tune 70B models on a single GPU.

What breaks, and how to choose

Quantization is not free, and the marketing tends to skip this part. Perplexity rises slightly at 8-bit and noticeably at 4-bit. Long-context reasoning, arithmetic and code generation degrade before casual conversation does, because those tasks depend on small differences between token probabilities that rounding erases. A quantized model rarely becomes obviously broken. It becomes subtly worse in ways that a benchmark average hides and a user notices in week three.

My rule of thumb: 8-bit is close to free and you should default to it. 4-bit is a genuine trade you should test on your own task, not on someone's leaderboard screenshot. Below 4-bit, assume damage until proven otherwise.

Also worth being honest about speed. Quantization reliably reduces memory. It only speeds things up when your hardware has kernels for that integer format, and when you were memory-bandwidth bound to begin with, which for single-user LLM inference you almost always are. A 4-bit model on a GPU with no INT4 kernel can be slower than FP16, because it dequantizes on the fly.

The honest way to learn this is to take one model, quantize it three ways, and run your own evaluation set against each. Two hours of that teaches more than any article. If you would rather do it with someone who has already broken things in production watching over your shoulder, the mentor-led sessions in the AI Masterclass track are built around exactly this kind of hands-on implementation work rather than slides.

Quantization is what turned large models from a data centre product into something that runs on the laptop you already own. Understanding it is the difference between downloading a Q4 file and hoping, and knowing what you traded away.

FAQs

1. Does quantization make a model faster or only smaller?

It always makes the model smaller, but speed depends on hardware support. You get real speedups when the chip has native kernels for that format, such as INT8 tensor cores, and when inference was limited by memory bandwidth rather than compute. Without matching kernels, a 4-bit model can run slower than FP16 because weights must be unpacked back to floats on every forward pass.

2. What do Q4_K_M and Q8_0 mean in GGUF filenames?

The number is the bit width, so Q4 is roughly four bits per weight and Q8 is eight. The suffix describes the scheme: _0 is a simple uniform method, while _K_M is a mixed approach that stores some layers at higher precision and represents a medium quality tier. Q4_K_M is the most common default because it balances size against quality well.

3. Is quantization the same as pruning or distillation?

No. Quantization keeps every parameter and stores each one in fewer bits. Pruning deletes parameters or whole channels, and distillation trains a smaller model to imitate a larger one. They are complementary and often stacked together in edge deployments.

4. Can I fine-tune a quantized model?

Not directly, because integer weights cannot receive gradient updates cleanly. QLoRA solves this by freezing the 4-bit base model and training small low-rank adapter layers in 16-bit on top of it, which is how people fine-tune very large models on a single consumer GPU.

5. Do I need a GPU to quantize a model?

For GGUF conversion with llama.cpp, no, a CPU with enough RAM to load the model is enough. Methods like GPTQ and AWQ run calibration passes through the network and are far more practical on a GPU, though they are usually a one-time job measured in minutes to a couple of hours.

6. Which quantization level should I use for a local LLM?

Start at Q4_K_M or 4-bit AWQ if you are memory constrained, since it fits most 7B to 8B models into about 5 GB. Move up to 8-bit if your task involves code, maths or long-context reasoning, where 4-bit degradation shows up first. Always test on your own prompts before committing.

Related reading

• The AI Learning Process, Explained Step by Step
• AI Learning Rate: What It Is, How to Pick One, and Why Training Fails Without It
• Building AI (Elements of AI): What the Course Actually Teaches, and What It Leaves Out
• A Neural Approach to Automated Essay Scoring: What the Paper Says and How to Build One