Obfuscation undercover: Unraveling the impact of obfuscation layering on structural code patterns

Abstract

Malware often uses code obfuscation to evade detection, employing techniques such as packing, virtualization, and data encoding or encryption. Despite widespread application, the impact of combining these techniques in a particular order – so-called obfuscation layering – on code analysis remains poorly understood. This study advances previous research by examining the effects of obfuscation layering on the classification of obfuscation techniques contained in binary code, focusing on how different layering combinations alter structural code patterns. Utilizing a dataset of 85 C programs modified with various combinations of code obfuscation techniques, we analyze the impact of obfuscation layering on structural code metrics such as its control flow complexity. Our study demonstrates that obfuscation layering significantly affects the ability to classify obfuscated code and that the order of applied obfuscations is less significant for classification than previously assumed. Through explainability methodologies our work offers novel insights for malware analysts and researchers to improve their detection strategies.