What is the actual difference?
Fine-tuning takes a model you already have and continues training it on data you supply, so the knowledge comes from your examples. Distillation trains a different, usually smaller model to reproduce the behaviour of a larger one, so the knowledge comes from the teacher's output rather than from labelled ground truth. The two are often discussed as alternatives because they can produce a similar end state — a small model that handles one job well — but they get there from opposite directions and they fail differently.
The practical consequence is where the cost sits. Fine-tuning needs labelled data and comparatively little compute. Distillation needs comparatively little labelling and a lot of generated data from a larger system. Which constraint binds for you is usually the whole decision.
When is fine-tuning the right answer?
When the task is narrow, the volume is high, and you can produce labels that look like production traffic. For classification and extraction the economics reversed back in favour of fine-tuning: a small fine-tuned model now beats a large prompted one on cost by an order of magnitude at comparable accuracy. Prompting still wins the general case, and nothing here argues otherwise.
The blocker has moved. It is no longer training cost, which has fallen; it is having a labelled set that reflects what actually arrives in production. That set is the least fashionable and most decisive asset a team can own, and it cannot be bought in a form that matches your traffic.
When is distillation the right answer?
When you serve enough volume to amortise the training. Distillation moves cost from inference to training, which suits anyone running high and steady throughput and suits nobody serving a long tail of low-traffic tasks, where the training will never be repaid. The releases that changed deployment decisions were mid-size models trained largely on generated data from larger systems, reaching accuracy that would have required an order of magnitude more parameters two years earlier.
The cost that is easy to overlook is legal rather than computational: training on generated output makes the provenance of the training data harder to establish, which matters more now that licensing terms are less simple than they were.
| Question | Points to fine-tuning | Points to distillation |
|---|---|---|
| What do you have? | Labelled data that matches production traffic | Access to a stronger model and budget to generate from it |
| Where can you afford cost? | Training is cheap; inference stays as it was | Training is expensive; inference gets cheaper |
| What does the traffic look like? | Narrow and high-volume, or narrow and modest | High and steady enough to repay the training |
| Long tail of small tasks? | Workable, one model per task | Rarely repays the training |
| Provenance of training data | Yours, and traceable | Generated, and harder to establish |
| Main failure mode | Forgetting what the model could already do | Inheriting the teacher's errors with no ground truth to correct them |
Can you do both at once?
That is what self-distillation fine-tuning is. The model learns from a version of itself conditioned on demonstrations, which reduces forgetting compared with training on the demonstrations directly. It is the standard answer to the main risk of fine-tuning — teaching a new skill while damaging the ones already there.
A paper we covered found an unexamined assumption inside it. Self-distillation always distils toward the full demonstration-conditioned teacher, so teacher influence is fixed at that endpoint and nothing controls how much of the demonstration transfers at any particular point in the sequence. Treating the teacher as a budgeted source instead — selecting, at each token, the distribution closest to the current student that still satisfies a constraint on how much teacher information may enter — improved on ordinary self-distillation in seven of eight model-task combinations across four backbones and two specialisation tasks, and matched it in the eighth.
The retention figures are the interesting part, because retention and specialisation normally trade against each other. In that work 73% of evaluations stayed within half a point of the base model, against 52% for the strongest baseline, and the same method produced the largest mean improvement across ten further benchmarks in mathematics, coding and competition mathematics. A method that improves both sides of a trade-off is usually describing something structural rather than a tuning win.
The underlying finding is worth separating from the method: when teacher information arrives matters as much as how much of it does. The standard formulation does not expose that as a knob, which is why nobody was turning it.
What neither of them fixes
Where the work physically runs. Mixture-of-experts serving changed which tensors are active per token and therefore which parts of a cluster must sit close together, and a model with 816 billion parameters of which about 42 billion are active per token is a different serving problem from a dense model of either size. A smaller model obtained by distillation or fine-tuning still has to be placed, cached and scheduled, and those decisions can cost more than the training did.
Questions
- What is the difference between fine-tuning and distillation?
- Fine-tuning continues training an existing model on your labelled data, so the knowledge comes from your examples. Distillation trains a smaller model to reproduce a larger one's output, so the knowledge comes from the teacher. Fine-tuning needs labels and little compute; distillation needs little labelling and a lot of generated data.
- Which is cheaper?
- They move the cost rather than removing it. Fine-tuning keeps training cheap and leaves inference as it was. Distillation makes training expensive and inference cheaper, which repays only where volume is high and steady.
- Can you use both together?
- Yes, and self-distillation fine-tuning is exactly that: the model learns from a version of itself conditioned on demonstrations, which reduces forgetting. Constraining how much teacher information enters at each token improved on it in seven of eight model-task combinations.
- What is the main risk of fine-tuning?
- Forgetting. Teaching a new skill can damage the ones the model already had, which is the problem self-distillation was introduced to reduce. The retention figures are the number to ask for, not just the task score.