Beyond the sandbox: Leveraging symbolic execution for evasive malware classification

Threat actors continuously update their code to incorporate counter-analysis techniques designed to evade detection and hinder the blocking of their malware. The first line of defence for malware authors is often to bypass static analysis, a relatively straightforward task using readily available tools such as packers and cryptors. To address this shortcoming, defenders send potential malware samples for execution in a sandbox environment. While sandboxing can provide valuable insights into the behaviour of software on an information system, advanced techniques like anti-virtualisation and hooking evasion allow malware to escape detection. The primary objective of this work is to complement sandbox execution with symbolic execution frameworks to detect new malware strains efficiently. Symbolic execution offers a distinct advantage over sandboxing by achieving greater coverage of all possible execution traces, as it can explore every potential execution path, regardless of the evasion methods employed by the malware authors. By carefully selecting the samples to be analysed, we can significantly reduce the workload while extracting essential dynamic features in a fraction of the time and with far fewer computational resources compared to sandboxing. To this end, we leverage machine learning in an automated pipeline, enabling the accurate detection of sophisticated malware using a real-world dataset. Our approach yields average F1 scores of 0.93 for the benign class and 0.99 for the malware class in a binary classification setup, surpassing the detection rates reported in the literature. Additionally, our method outperforms a commercial malware sandbox when applied to the same dataset, further highlighting the efficacy of the proposed method.