Knowledge Distillation

Knowledge distillation transfers useful behavior from a large teacher model into a smaller student model. The student learns from the teacher’s probability distributions, generated examples, internal representations, or a combination of those signals and original labels. A distilled model may run faster, cost less, and fit on hardware that cannot host the teacher, making the technique valuable for mobile and edge deployment. Compression can reduce accuracy or remove rare capabilities, and the student may inherit the teacher’s bias, errors, or unsafe behavior. Teams compare performance across important subgroups and tasks, verify latency and memory gains, and confirm that legal or licensing terms permit the training process.