ModiFedCat: A multi-modal distillation based federated catalytic framework

Abstract

The integration of multi-modal data in federated learning systems faces significant challenges in balancing privacy preservation with effective cross-modal correlation learning under strict client isolation constraints. We propose ModiFedCat, a novel curriculum-guided multi-modal federated distillation framework that combines hierarchical knowledge transfer with adaptive training scheduling to enhance client-side model performance while maintaining rigorous data privacy. Our method computes multi-modal knowledge distillation losses at both the feature extraction and output layers, ensuring that local models are consistently aligned with the global model throughout training. Additionally, we introduce a unique catalyst strategy that dynamically schedules the integration of the distillation loss. By initially training the global model without distillation, we determine the optimal timing for its introduction, thereby maximizing the effectiveness of knowledge transfer once local models have stabilized. Experimental results on three benchmark datasets, AV-MNIST, MM-IMDB, and MIMIC III, demonstrate that ModiFedCat outperforms existing multi-modal federated learning methods. The proposed framework significantly improves the fusion capability of multi-modal models while maintaining client data privacy. This approach balances local adaptation and global knowledge integration, making it a robust solution for multi-modal federated learning scenarios.