Enhancing knowledge distillation via genetic recombination

Abstract

Diverging from conventional knowledge distillation methods that solely emphasize improving the utilization of the teacher’s knowledge, this paper explores the generation of stronger student models within available knowledge. We first conceptualize the knowledge distillation process as a genetic evolution model. The student model is regarded as an independent individual, with its parameters representing the genes of that individual. These genes are partitioned into several alleles according to the architecture of the student model. Following that, we propose a universal strategy to enhance existing knowledge distillation methods by introducing genetic recombination. Prior to distillation, we initialize two independent identically distributed student models with different random seeds to obtain the first generation of genes. With each epoch of distillation, these genes evolve into the next generation. At specific generations, we randomly select one exchangeable allele from each of the two students for exchange. Our focus lies in determining the alleles to exchange and their corresponding exchange frequency (i.e., crossing-over value). This approach provides more choices and possibilities for subsequent evolution. Extensive experiments confirm the effectiveness of the strategy, demonstrating improvements across 12 distillation methods and 17 teacher–student combinations.