Multiteacher Multifeature Knowledge Distillation Based on Dynamic Mutual Selection

Abstract

Multiteacher knowledge distillation methods exhibit superior adaptability and flexibility over single-teacher models when handling diverse datasets. However, these methods encounter significant challenges when dealing with varying feature types and different numbers of teacher models. The first challenge lies in selecting the optimal distillation samples for the student model, and the second involves choosing the most appropriate teacher model for specific feature types. To address these challenges, we propose a multiteacher multifeature knowledge distillation learning framework based on dynamic mutual selection (DMS-MTMF). DMS-MTMF employs a dynamic mutual selection module search for the optimal combination of teacher models and features, ensuring a harmonious collaboration that yields superior training outcomes. During this process, selection mismatches may arise between teacher models and features. To address this, we adopt the multiagent depth determination policy gradient algorithm for collaborative optimization within the mutual selection module. Finally, a dynamic sample selection module is proposed to model the sample selection cost. DMS-MTMF eliminates the distortion caused by static distillation sample selection. Experimental results on the HHU and PU datasets not only highlight the importance of dynamic mutual selection and sample selection but also achieve higher accuracy than other comparative methods.