AI Learning
What Is Quantization? The Plain Answer, From Audio CDs to 4-Bit LLMs
Quantization means representing values with fewer, coarser levels. In AI it means storing a model's weights in 8 or 4 bits instead of 32, which shrinks the file, speeds up inference and costs you a little accuracy. This explains the idea, the arithmetic behind it, where it breaks, and how to read names like Q4_K_M.
CSEWhy ·
Quantization means representing values with fewer, coarser levels. In AI it means storing a model's weights in 8 or 4 bits instead of 32, which shrinks the file, speeds up inference and costs you a little accuracy. This explains the idea, the arithmetic behind it, where it breaks, and how to read names like Q4_K_M.
The short answer
Quantization is the process of taking values from a large or high-precision range and mapping them onto a smaller set of discrete levels. Rounding every price in a shop to the nearest rupee is quantization. So is storing a neural network's weights as 8-bit integers instead of 32-bit floating point numbers.
In AI, that second meaning is what people usually mean. A model's weights are just millions or billions of numbers. Store each one in 32 bits and a 7-billion-parameter model occupies roughly 28 GB. Store each one in 4 bits and the same model fits in about 3.5 GB, which is the difference between needing a data centre GPU and running it on a laptop. You pay for that with a small loss of precision, because each weight is now snapped to one of 16 possible levels instead of billions.
That trade is the whole subject. Everything else is detail about how to make the loss small.
| Precision | Bits per weight | Approx. size of a 7B model | Typically used for |
|---|---|---|---|
| FP32 | 32 | 28 GB | Training, reference accuracy |
| FP16 / BF16 | 16 | 14 GB | Standard GPU inference |
| INT8 | 8 | 7 GB | Server inference, mobile NPUs |
| INT4 | 4 | 3.5 GB | Laptops, consumer GPUs, phones |
| INT2 | 2 | 1.8 GB | Research, usually degraded badly |
The idea is older than deep learning
If you searched this term for a signal processing course rather than an AI one, the core concept is identical. An analogue-to-digital converter samples a continuous voltage and has to round it to one of a fixed number of levels. CD audio uses 16 bits, which gives 65,536 possible amplitude values per sample. The gap between the true value and the stored one is quantization error, and in audio you hear it as noise.
JPEG does the same thing deliberately. After the image is transformed into frequency components, a quantization table divides away the high-frequency detail your eye barely notices. That is why the quality slider exists, and why a heavily compressed photo develops blocky edges.
Same principle, different victim. In audio you lose dynamic range, in JPEG you lose detail, in a neural network you lose a bit of the model's judgement.
How a model actually gets quantized
Take one layer of a network. Its weights might all sit between -0.82 and 0.91. To store them as 8-bit signed integers you need to map that real range onto the integers -128 to 127. You compute a scale factor (roughly 0.0068 here) and sometimes a zero point to handle asymmetric ranges. Each weight becomes round(weight / scale). At inference time you multiply back by the scale.
Doing this once per tensor is cheap and crude. Doing it per output channel is better, because one channel with a wide range no longer forces a coarse step size on every other channel. Activations are harder than weights, because you do not know their range until data flows through, so you run a few hundred sample inputs through the model first and record the ranges. That step is called calibration.
The real enemy is outliers. In large language models, a small number of activation dimensions carry values tens of times larger than everything else, and they appear reliably once models pass roughly 6 billion parameters. One extreme value stretches the range and makes every normal value round to almost the same integer. Modern methods handle this by keeping the outlier dimensions in higher precision, or by shifting difficulty from activations into weights before quantizing.
Two ways to do it, and what each costs you
Post-training quantization takes a finished model and compresses it. It needs no labelled data, runs in minutes to a couple of hours, and for most models at 8 bits the accuracy difference is too small to notice. Push to 4 bits and good methods still land within about one percent on most benchmarks. Push to 2 bits and output quality usually collapses.
Quantization-aware training simulates the rounding during training, so the model learns weights that survive it. It costs a full training or fine-tuning run, and it is what you reach for when 8-bit post-training quantization still hurts, typically on small models or tight edge hardware where there is no spare capacity to absorb the error.
| Post-training quantization | Quantization-aware training | |
|---|---|---|
| What it needs | A trained model, a small calibration set | Full training pipeline and data |
| Time | Minutes to hours | Hours to days |
| Typical accuracy loss at INT8 | Under 1% | Often near zero |
| Best for | LLMs, large vision models | Small models, phones, microcontrollers |
What this means if you are the one deploying it
If you have downloaded a local model, you have already met quantization under names like Q4_K_M, Q5_K_S or Q8_0. The number is bits per weight. The letter suffix describes how mixed the precision is across layers, with K-quants keeping the sensitive parts (attention and the output layer) at higher precision than the rest. For most people running a model on a 16 GB laptop, a 4-bit K-quant of a larger model beats an 8-bit version of a smaller one. More parameters at lower precision usually wins over fewer parameters at higher precision.
On GPUs, GPTQ and AWQ are the common weight-only 4-bit methods, and bitsandbytes gives you quick 8-bit and 4-bit loading inside Hugging Face. QLoRA goes further: it fine-tunes adapters on top of a frozen 4-bit base, which is how people train 13B models on a single consumer card.
The gap between reading this and being useful with it is one afternoon of actually loading a model two ways and measuring the difference in memory, tokens per second and output quality. If you want that kind of hands-on build time with people reviewing your work rather than a reading list, that is the shape of the AI Creator Fellowship, an eight-week programme built around shipping projects.
One last thing worth internalising. Quantization is not a trick that makes models worse in some hidden way, it is a deliberate choice about where to spend precision. Spend it where the model is sensitive, save it everywhere else.
FAQs
1. Does quantization make a model less accurate?
Slightly, and usually less than people expect. Eight-bit quantization of a large model typically costs under one percent on standard benchmarks, and good four-bit methods stay close behind. Below three bits, quality degrades sharply.
2. What is the difference between quantization, pruning and distillation?
Quantization keeps every parameter but stores each one in fewer bits. Pruning deletes parameters or whole channels entirely. Distillation trains a smaller new model to imitate a larger one. They are independent and can be combined.
3. What does Q4_K_M mean in a GGUF model file?
Q4 means roughly four bits per weight, K refers to the K-quant scheme that uses block-wise scales, and M is the medium size variant which keeps some sensitive layers at higher precision. Q4_K_M is the usual default for local inference because it balances size against quality well.
4. Can you fine-tune a quantized model?
Yes, using QLoRA. The base model stays frozen in 4-bit while small low-rank adapter layers are trained in higher precision, which lets you fine-tune models of 7B to 13B parameters on a single consumer GPU.
5. Does quantization always make inference faster?
Not automatically. You get the memory saving everywhere, but the speed gain depends on whether your hardware has native integer or low-bit kernels. On a CPU or a GPU without INT4 support, dequantizing on the fly can eat part of the benefit.
6. Is quantization in signal processing the same as in machine learning?
The underlying mathematics is the same: mapping a continuous or high-precision range onto a finite set of levels and accepting rounding error. The difference is what you are protecting, audio fidelity in one case and model predictions in the other.