secPEFL: Strengthening federated learning security for Portable Executable malware detection in distributed networks

Traditional centralized malware detection approaches are increasingly vulnerable to privacy risks and data breaches, particularly given stringent regulatory requirements. To address these challenges, we propose a Federated learning-based system for malware classification on PE executable files, emphasizing enhanced data privacy and security. Our approach leverages a Convolutional Neural Network architecture with two modules: a detection module for detecting malicious files and a classification module for identifying malware types and supporting defense strategies. The system operates on grayscale images and incorporates advanced security measures, including Secure Sockets Layer for secure communication, InterPlanetary File System for distributed storage, and Local Differential Privacy to counter inference attacks. The proposed system mitigates Sybil attacks through a participant selection mechanism based on reputation history stored on the blockchain network. The blockchain is also used as a reward platform for contributors, utilizing a Shapley value-based reward mechanism from game theory. Experimental results show that the proposed system delivers superior malware classification performance while maintaining security aspects. The highest accuracy achieved is 96.32% on IID (Independent and Identically Distributed) data and 88.06% on non-IID data for malware classification tasks. The experiments also reveal that as the level of noise added using Differential Privacy increases, security improves, but the model’s accuracy decreases correspondingly.