Knowledge distillation is the process of transfering knowledge from a large model to a smaller model. Smaller model are necessary for less powerful hardware like mobile, edge devices. There are two parts of knowledge distilation.

  • Teacher model(large model)
  • Student model(distil model)

kd

Knowledge Distillation are three types:

  • Response based distillation
  • Feature based distillation
  • Relational based distillation

For more information check out the resources below:

Papers

Blog

Repositories(codes)