ConvNeXt_GHSA: Integrating hybrid gated attention for malware image classification

Malware classification based on image representation has emerged as an effective approach to enhancing security systems against evolving threats. However, challenges such as suboptimal feature extraction, insufficient adaptive attention fusion, and class imbalance remain unresolved. To address these issues, this paper proposes a deep learning-based classification framework named ConvNeXt_GHSA. The model is built upon a pretrained ConvNeXt backbone and incorporates a novel Gated Hybrid Self-Attention (GHSA) mechanism, which integrates channel, local, and global attention branches to capture multi-scale, discriminative features. At gating strategy is employed to adaptively fuse information from the three branches according to their contextual relevance. Additionally, Focal Loss and label smoothing are adopted during training to alleviate the impact of class imbalance and enhance minority class recognition. Experimental evaluations on three public malware image datasets—Malimg, MaleVis, and Dumpware10—demonstrate that ConvNeXt_GHSA achieves classification accuracies of 99.79%, 99.23%, and 99.78%, respectively. These results confirm the proposed model's robustness, effectiveness, and generalization ability in malware image classification tasks.