FORCE: Federated Orthogonality-Aware Low-Rank Adaptation for Communication-Efficient Fine-Tuning

Abstract

Federated fine-tuning of Large Language Models (LLMs) is fundamentally bottlenecked by the uplink communication capacity of edge networks. While Parameter-Efficient Fine-Tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) reduce the payload by representing a dense $d\times k$ update with rank- $r$ factors, they may suffer from subspace interference when aggregated across highly heterogeneous clients. In this letter, we propose FORCE, a communication-efficient federated adaptation framework. Inspired by the philosophy of magnitude-direction decoupling, FORCE introduces a local “project-and-compress” geometric heuristic. Rather than transmitting dense orthogonal matrices, clients apply a lightweight 1-step gradient correction to align their low-rank updates toward an approximate orthonormal target before server aggregation, while preserving the $\mathcal {O}(r(d+k))$ low-rank communication budget. Empirical results demonstrate that FORCE safely preserves model capacity in balanced networks and significantly mitigates performance degradation.