Deep hybrid approach with sequential feature extraction and classification for robust malware detection

Malware attacks have escalated significantly with an increase in the number of internet users and connected devices. With the increasingly different types of malware released by hackers, designing new and competitive techniques to detect advanced malware is essential. In the proposed research, we have developed a multi-level feature extraction technique using deep learning architectures and a classification model to classify malware families. The essential features from the malware images are extracted using the Gated Recurrent Unit in the first step, which are further fed to a Convolutional Neural Network model for extracting the final feature vector. The multi-level feature selection is followed by classification into various malware families using Cost-sensitive Boot Strapped Weighted Random Forest (CSBW-RF). The proposed approach gave promising results of 99.58 % accuracy in distinguishing the 25 different malware families on the Mallmg dataset. This hybrid model gave significantly better performance scores for classifying visually similar malware families. The generalizability of the proposed model is benchmarked with the popular Microsoft Big 2015 dataset and has achieved comparatively higher performance scores than many existing models. This benchmarking demonstrates the robustness and scalability of our approach. The use of cost-sensitive learning and bootstrapping techniques also contributed to the model’s ability to generalize well to new and unseen data. These enhancements ensure that our model can be effectively applied in diverse real-world scenarios, maintaining high performance across different environments and malware types. This research can contribute to detecting malware attacks and can be integrated in threat monitoring systems. The successful application of this hybrid model indicates its potential for deployment in real-world cybersecurity environments, providing a strong defense against evolving malware threats.