A prototype-based framework for open-set heterogeneous federated face recognition

Abstract

Federated learning enables privacy-preserving face recognition by keeping raw images on users’ devices, addressing challenges of non-IID identity distributions and heterogeneous model architectures. To tackle these, we propose FedPFR, a prototype-based, architecture-agnostic framework for open-set Heterogeneous Federated Face Recognition (HtFFR). Each client computes identity prototypes-mean feature embeddings of local classes-that are uploaded to the server and redistributed without averaging, while local models are trained with a hybrid loss combining CosFace and a novel prototype-anchor contrastive (PAC) loss. This design preserves semantic integrity by reducing the distance between local embeddings and their global prototypes and enlarging inter-class separation. We provide a mathematical convergence analysis proving that FedPFR converges to a stationary point under appropriate learning conditions. Extensive experiments on the IJB-C benchmark with 20 heterogeneous clients show that FedPFR achieves strong verification performance, with 23.39 % TAR at FAR=1e-6, outperforming local-only training (21.70 %) and prior heterogeneous FL baselines. Furthermore, our cost analysis quantifies the computation, communication, and storage overhead, confirming the framework’s scalability and practicality. Ablation studies and clustering analyses further demonstrate that FedPFR produces compact and discriminative embeddings, highlighting its robustness as a resource-efficient solution for real-world open-set federated face recognition.