MalAder: Decision-Based Black-Box Attack Against API Sequence Based Malware Detectors

Abstract

The API call sequence based malware detectors have proven to be promising, especially when incorporated with deep neural networks (DNNs). Several adversarial attack methods are proposed to fool these detectors by introducing undetectable perturbations into normal samples. However, in real-world scenarios, the malware detector provides only the predicted label for a given sample, without exposing its network architecture or output probability, making it challenging for adversarial attacks under the decision-based black-box. Existing work in this area typically relies on random-based methods that suffer high costs and low attack success rates. To address these limitations, we propose a novel decision-based black-box attack against API sequence based malware detectors, called MalAder. Our approach aims to improve the attack success rate as well as query efficiency through a directional perturbation algorithm. First, it utilizes attention-based API ranking to assess the importance of API calls in the context of different API sequences. This assessment guides the insertion position for perturbation. Then, the perturbation is carried out using benign distance perturbing, which gradually shortens the semantic distance from adversarial API sequences to a set of benign samples. Finally, our algorithm iteratively generates adversarial malware samples by performing perturbations. In addition, we have implemented MalAder and evaluated its performance against two classic malware detectors. The results show that MalAder outperforms state-of-the-art decision-based black-box adversarial attacks, proving its effectiveness.