The reason to make a model smaller is rarely curiosity. It is that a model small enough to serve cheaply, or to run where there is no GPU at all, is worth more in most deployments than one that is marginally better and ten times the cost per token. There are two established ways to get there and they are routinely confused, including by people choosing between them.
What each technique actually does
| Distillation | Quantisation | |
|---|---|---|
| What happens | A smaller model is trained to match a larger one's outputs | The weights of an existing model are stored at lower precision |
| What you end up with | A different model | The same model, approximated |
| What transfers | Behaviour, including the shape of the teacher's uncertainty | Nothing is transferred; something is lost |
| How it fails | The student behaves differently from the teacher on cases nobody tested | Accuracy degrades unevenly across tasks |
| How to evaluate it | As a new model, on your own tasks | Against the full-precision original, task by task |
The distinction matters for a practical reason: distillation is not compression of a file, it is retraining. Teams that treat a distilled model as a drop-in replacement and run the old evaluation suite are measuring the wrong thing, because what they have is a different model that happens to be cheaper.
Why doing both at once beats doing them in sequence
The usual pipeline finishes a model and then quantises it, so the precision loss is absorbed after training, where nothing can compensate for it. Quantisation-aware distillation inverts that: the lower-precision student is trained against the full-precision teacher, so the loss is absorbed during training.
The published recovery figures against a BF16 baseline were 97.1% at 230M parameters, 96.5% at 350M, 97.4% at 1.2B and 96.6% at 2.6B. But recovering accuracy against full precision is what any quantisation method claims. The useful comparison is against the alternatives: those four-bit checkpoints matched what larger quantisation formats achieve while decoding 3 to 33% faster. A smaller, cruder quantisation performing like a bigger one is a straight saving in both memory and time.
Note what the evaluation covered, because it is the part usually skipped: reasoning, instruction following, tool use and agentic behaviour across six benchmarks, with speed measured on a laptop, a mini PC, a phone and a Raspberry Pi. A small model that reasons well and stops following instructions is not a usable small model, and only a suite that tests both will catch it.
When distillation goes wrong
Teaching a model a new skill without damaging the ones it already has is the practical problem underneath most of this work. Self-distillation addresses it by having a model learn from a version of itself conditioned on demonstrations, which forgets less than training on the demonstrations directly. Recent work points at an assumption inside that method: it always distils toward the full demonstration-conditioned teacher, with no control over how much teacher information enters at any particular point.
Treating the teacher as a budgeted source rather than a target improved on ordinary self-distillation in seven of eight model-task combinations and matched it in the eighth. The retention gap was larger: 73% of evaluations stayed within half a point of the base model, against 52% for the strongest baseline. Retention and specialisation usually trade against each other, so a method improving both is describing something structural rather than a tuning win.
Where the formats actually live
For local inference the practical constraint is not theory but tooling. GGUF has become the de facto format: it packages weights, tokenizer information and an optional chat template in a single file, supports a range of quantisation levels, and powers the common local runtimes. Support for loading it has spread into the larger training libraries by reusing the original kernels rather than reimplementing the arithmetic — which is the difference between compatibility that is a checkbox and compatibility that is usable.
Choosing between them
- If the constraint is memory on the device, quantisation moves the number directly, and quantisation-aware training recovers most of what it costs.
- If the constraint is cost per token at scale, a distilled model changes the economics more than a quantised large one.
- If the task is narrow and repetitive, distillation into a small model is usually the larger win, because generality is what you are paying for and not using.
- In either case, evaluate instruction following and tool calling separately from reasoning: the degradations do not arrive together.
- Check the licence on the weights before anything ships inside someone else's product. It is not always stated in the announcement.
Questions
- What is the difference between distillation and quantisation?
- Distillation retrains a smaller model to reproduce a larger one's behaviour, producing a different model. Quantisation stores an existing model's weights at lower precision, producing an approximation of the same model. One is retraining, the other is a lossy transformation.
- How much accuracy does quantisation cost?
- Less than it used to, if the quantisation is accounted for during training. Quantisation-aware distillation checkpoints recovered 96.5% to 97.4% of full-precision accuracy at sizes from 230M to 2.6B parameters, while decoding 3 to 33% faster than the quantisation formats they replace.
- Can a distilled model reuse the original's evaluation suite?
- It should not be trusted to. Distillation produces a different model that behaves similarly, so it has to be evaluated as a new model — particularly on instruction following and tool use, which degrade separately from reasoning.
- Does a smaller model still follow instructions and call tools?
- That is exactly what has to be tested rather than assumed. The evaluation behind the figures above covered reasoning, instruction following, tool use and agentic behaviour precisely because a model can hold its reasoning score and lose the rest.