Embracing knowledge integration from the vision-language model for federated domain generalization on multi-source fused data

Abstract

Federated Domain Generalization (FedDG) has attracted attention for its potential to enable privacy-preserving fusion of multi-source data. It aims to develop a global model in a distributed manner that generalizes to unseen clients. However, it faces the challenge of the tradeoff between inter-client and intra-client domain shifts. Knowledge distillation from the vision-language model may address this challenge by transferring its zero-shot generalization ability to client models. However, it may suffer from distribution discrepancies between the pretraining data of the vision-language model and the downstream data. Although pre-distillation fine-tuning may alleviate this issue in centralized settings, it may not be compatible with FedDG. In this paper, we introduce an in-distillation selective adaptation framework for FedDG. It selectively fine-tunes unreliable outputs while directly distilling reliable ones from the vision-language model, effectively using knowledge distillation to address the challenge in FedDG. Furthermore, we propose a federated energy-driven reliability appraisal (FedReap) method to support this framework by appraising the reliability of outputs from the vision-language model. It includes hypersphere-constraint energy construction and label-guided energy partition. These two processes enable FedReap to acquire reliable and unreliable outputs for direct distillation and adaptation. In addition, FedReap employs a dual-level distillation strategy and a dual-stage adaptation strategy for distillation and adaptation. Extensive experiments on five datasets demonstrate the effectiveness of FedReap compared to twelve state-of-the-art methods.