Model Distillation

Short answer

Model distillation is a technique for training a smaller student model to mimic the outputs of a larger teacher model. The student learns to approximate the teacher's behavior, delivering similar quality on targeted tasks at lower cost and latency. It lets companies run capable AI features more cheaply and sometimes directly on devices.