Model Distillation
Model distillation, also known as knowledge distillation, is a process where a smaller, 'student' model learns to mimic the behavior of a larger, more complex 'teacher' model. Instead of training on hard labels (e.g., Cat or Dog), the student learns from the teacher's 'soft targets,' which are probability distributions over classes, providing richer information. This allows the student model to achieve comparable performance to the teacher while being significantly more efficient.
It's like a seasoned mentor (teacher) sharing their nuanced understanding with a junior colleague (student), who then performs almost as well but with less experience.
It enables deploying high-performing models to resource-constrained environments like mobile devices or embedded systems.
Learn one new AI thing every day.
Daily Deck sends you seven plain-English cards like this every morning. Free.
Start free