HyperMem: Hypernetwork with memory for forgetting problem in federated reinforcement learning

Abstract

Federated reinforcement learning plays a crucial role in decentralized and privacy-preserving policy optimization but is challenged by task heterogeneity and client dropout. Several approaches proposed for these issues, but few consider their combined impact. In this paper, we reveal the catastrophic forgetting phenomenon arising from their coexistence, which significantly degrades the global model’s performance on offline clients. We formally define this forgetting problem and establish an exponential convergence rate for hypernetwork-based federated learning methods, highlighting the adverse effects of embedding length on forgetting. Furthermore, we demonstrate the equivalence between the mean squared error loss and the chain rule in hypernetwork updates, introducing a novel updating paradigm. Based on our theoretical insights, we propose HyperMem with three key components: (1) Constrained Principal Component Embedding, which limits embedding length and enhances hypernetwork priors; (2) In-cluster and Out-cluster Losses, designed under the new updating paradigm to dynamically select fitting targets and mitigate, or even resolve, forgetting problems; and (3) Adapter Pool, enabling federated training of structurally heterogeneous client models caused by greater task heterogeneity. Comprehensive experiments demonstrate that HyperMem effectively overcomes the forgetting problem, improving training performance by 14.95 % compared to state-of-the-art methods. We implemented HyperMem as a pluggable Spark service for practical applications, reducing job runtime by 42.38 % and communication costs by 98.6 %, while ensuring data security.