Boosting Federated Domain Generalization: Understanding the Role of Advanced Pretrained Architectures

Abstract

Federated learning (FL) enables privacy-preserving model training across decentralized data. However, significant data heterogeneity, common in domains like the Internet of Things (IoT), hinders generalization. Federated domain generalization (FDG) extends the FL paradigm by aiming to train models that generalize effectively to unseen domains, without requiring access to data from those domains during training. Current FDG methods primarily use residual network (ResNet) backbones pretrained on ImageNet-1K, limiting adaptability due to architectural constraints and limited pretraining diversity. This reliance has created a gap in leveraging advanced architectures and diverse pretraining datasets to address these challenges. To bridge this gap, we present the first comprehensive investigation into the efficacy of advanced pretrained architectures, such as vision transformers, ConvNeXt, and Swin Transformers, in enhancing FDG performance. Unlike ResNet, these architectures capture global context and long-range dependencies, making them well-suited for FDG. Beyond architectural evaluation, we systematically assess the impact of diverse pretraining datasets and compare self-supervised and supervised strategies. Our analysis rigorously investigates the influence of architectural depth, parameter efficiency, and the interplay between diverse model families and dataset characteristics on FDG performance. We find that advanced architectures pretrained on large datasets significantly outperform ResNet models. Specifically, ConvNeXt architectures outperform all other candidates. We find self-supervised methods using masked image patch reconstruction via discrete token prediction outperform their supervised counterparts. We observe that certain advanced model variants with fewer parameters outperform larger ResNet models. This underscores the need for advanced architectures and scalable pretraining to enable efficient and generalizable FDG.