FDBA: Feature-guided Defense against Byzantine and Adaptive attacks in Federated Learning

Federated Learning (FL) is a general paradigm that enables decentralized model training while preserving data privacy, allowing multiple clients to collaboratively train a global model without sharing raw data. With the increasing application of Large Language Models (LLMs) in fields like finance and healthcare, data privacy concerns have grown. Federated LLMs have emerged as a solution, enabling the collaborative improvement of LLMs while protecting sensitive data. However, federated LLMs, like other FL applications, are vulnerable to Byzantine attacks, where one or more malicious clients attempt to poison the global model by corrupting local data or sending crafted local model updates to the server. Existing defenses that focus on directly analyzing local updates struggle with the large parameter sizes of modern models like LLMs. Thus, we need to design more effective defense mechanisms that can scale to models of varying sizes.

In this work, we propose FDBA, a method designed to enhance robustness and efficiency in FL. Unlike traditional defenses that rely solely on analyzing local model updates, our approach extracts features called PDist from the models to describe the impact of these updates. We propose a cooperative learning mechanism based on PDist, which evaluates features across three dimensions to determine whether the update is malicious or benign. Specifically, FDBA first performs clustering on PDist, and then classifies the clustering results using an additional auxiliary data to efficiently and accurately identify malicious clients. Finally, historical information is leveraged to further enhance the accuracy of the detection. We conduct extensive evaluations on three datasets, and the results show that FDBA effectively defends against both existing Byzantine and adaptive attacks. For example, under six types of Byzantine attacks, FDBA maintains the same accuracy as the global model trained with FedAvg without any attacks. Additionally, we perform evaluations on LLMs, and the results demonstrate that FDBA still achieves high accuracy under representative Byzantine attacks.