Knowledge Distillation

Knowledge distillation is a technique where a smaller, simpler AI model, called the student, is trained to mimic the behavior of a larger, more complex model, the teacher. The goal is to transfer the teacher's 'knowledge' and achieve comparable performance with a more efficient student model, often by using the teacher's soft probabilities as targets.