HExNet: Enhancing malware classification through hierarchical CNNs and multi-level feature attribution

The ever-shifting landscape of malware presents a significant threat, as it routinely circumvents traditional defenses. This paper presents HExNet, a Hierarchical Explainable Convolutional Neural Network (CNN) architecture, designed to improve malware analysis and bolster security defenses. Recognizing the growing sophistication of malware, HExNet leverages a dual image representation, converting assembly mnemonics and raw bytecode of malware into visual representations for in-depth pattern recognition. The architecture, optimized for performance and security relevance, integrates multi-level features to enhance detection accuracy. To increase trust and facilitate security audits, HExNet incorporates SHAPley Additive Explanations (SHAP), Class Activation Maps (CAM), and GIST descriptors, providing transparent insights into the model’s classification process. t-SNE visualizations further demonstrate HExNet’s ability to effectively separate malware families, aiding in security intelligence. Evaluated on the Microsoft Malware Classification Challenge (BIG 2015) dataset, HExNet achieves an overall F1-score of 0.9890, with three malware families reaching a perfect F1-score of 1.0 and the remaining six families achieving near-optimal values. To evaluate the generalization capability, we further tested HExNet on a custom dataset consisting 26,401 samples collected from VirusShare, where the proposed model achieved an F1-score of 0.9724, demonstrating generalization performance across diverse malware datasets.