Quick answer: Knowledge distillation is a technique for training a small student model to reproduce the behavior of a larger, more capable teacher model, so the student ends up nearly as good at the task while being smaller, faster, and cheaper to run. Instead of learning only from the correct answer for each example, the student also learns from the teacher’s full output, which carries more information about how the teacher reasons through a problem. It is one of the main ways a slow, expensive model becomes a fast one that fits on a phone or answers instantly.
So what is knowledge distillation in practice? It is how a chatbot company turns a large flagship model into a cheap, fast variant with most of the same quality, and how a vision model that only runs well on a server becomes one that runs on a phone’s camera in real time. This entry covers how distillation actually works, where you meet it, how it differs from other ways of shrinking a model, and where it falls short.
What Is Knowledge Distillation?
The idea comes from a 2015 paper by Geoffrey Hinton and colleagues, which showed that a large trained model contains more useful information than just its final answers. When a teacher model classifies an image of a cat, its raw output is not a single label but a full probability spread across every possible label, including small, nonzero probabilities on labels that are wrong but related, such as dog or fox.
Hinton’s paper called this extra information dark knowledge. It reveals which mistakes the teacher considers plausible, which turns out to be useful training signal that a single correct label cannot provide on its own. Knowledge distillation is built around capturing that signal and using it to train a smaller model.
The result is a student model that was never trained on the original task’s full dataset in the traditional sense. It was trained to match a teacher that already learned that dataset, which is usually a faster and more data-efficient path to a small, capable model than training a small model from scratch.
It is easy to confuse with transfer learning, since both start from an already-trained model. Transfer learning adapts one model to a new task by continuing to train its own weights. Knowledge distillation trains a second, usually much smaller, model to imitate the first one’s behavior on the same task.
The teacher’s weights never move, and the student can even have a completely different architecture.
How Does Knowledge Distillation Work?
A trained teacher model generates predictions for a set of training examples. Rather than passing along only the final answer, the process uses what are called soft labels, the teacher’s full probability distribution over every possible output for a given example.
A setting called temperature controls how much of that distribution gets used. At normal settings, a confident model’s output is dominated by one very high probability and a lot of near-zero ones, which hides most of the dark knowledge. Raising the temperature flattens the distribution so the smaller, informative probabilities become large enough for the student to actually learn from.
The student model, usually with far fewer parameters than the teacher, then trains to match that softened output, typically alongside the original correct labels. It never needs to be as large as the teacher to get most of the way there, because it is learning a simplified version of a problem the teacher has already solved, rather than solving the raw problem from nothing.
Where Is Knowledge Distillation Used?
DistilBERT is the most-cited example. Hugging Face distilled Google’s BERT language model into a version 40 percent smaller, and the original DistilBERT research reported that it keeps 97 percent of BERT’s language understanding while running 60 percent faster.
Large language model providers use the same idea to build cheaper, faster variants of their flagship models. OpenAI’s own platform offers a formal distillation workflow for turning outputs from a larger model into training data for a smaller, cheaper one, which is the same mechanism behind cost-efficient model variants that keep most of a flagship model’s quality at a fraction of the price.
It also shows up in on-device AI. Apple’s on-device language model, part of Apple Intelligence, was trained using a distillation loss from a larger internal teacher model, part of the broader push to get useful language models running locally instead of over the network. That is where distillation, edge AI, and the hardware built for it meet: a distilled model still needs a chip suited to running neural networks efficiently, which is exactly what a phone’s NPU is built for.
Knowledge Distillation vs Pruning and Quantization: What’s the Difference?
All three are model compression techniques aimed at the same goal, a smaller and faster model, but they work on different parts of the problem. Pruning removes weights or entire channels from an already-trained model that contribute little to its output, shrinking it after the fact. Quantization keeps every parameter but represents each one with fewer bits, trading numeric precision for a smaller memory footprint and faster math.
Knowledge distillation is different in kind: it trains a new, smaller model from scratch, guided by a larger one, rather than trimming or re-encoding an existing model’s weights. It also differs from fine-tuning, which adjusts one model’s own weights for a narrower job; distillation produces an entirely separate model built to imitate the first.
In practice the three compression techniques are often combined, distilling a smaller architecture and then quantizing it for deployment, since each one attacks a different source of cost.
What Are the Limits of Knowledge Distillation?
A student model is bounded by its teacher. Distillation transfers what the teacher already knows; it does not add capability the teacher never had, and a student trained on a mediocre teacher will inherit that ceiling along with any of its blind spots or errors.
Training a good student also still requires running the teacher model, often repeatedly, across a large set of examples to generate soft labels, which adds real compute cost before the smaller, cheaper model even exists. And a student that is compressed too aggressively relative to the task’s real difficulty will show a visible quality gap, particularly on inputs unlike anything in the distillation dataset.
None of that erases the appeal. A well-distilled student model can deliver most of a much larger model’s quality at a small fraction of its size and cost, which is why distillation keeps showing up wherever a team needs a capable model that also has to run fast, cheap, or on a device with no server behind it.