A heuristic approach for detection of obfuscated malware

Abstract

Obfuscated malware has become popular because of pure benefits brought by obfuscation: low cost and readily availability of obfuscation tools accompanied with good result of evading signature based anti-virus detection as well as prevention of reverse engineer from understanding malwares' true nature. Regardless obfuscation methods, a malware must deobfuscate its core code back to clear executable machine code so that malicious portion will be executed. Thus, to analyze the obfuscation pattern before unpacking provide a chance for us to prevent malware from further execution. In this paper, we propose a heuristic detection approach that targets obfuscated windows binary files being loaded into memory - prior to execution. We perform a series of static check on binary file's PE structure for common traces of a packer or obfuscation, and gauge a binary's maliciousness with a simple risk rating mechanism. As a result, a newly created process, if flagged as possibly malicious by the static screening, will be prevented from further execution. This paper explores the foundation of this research, as well as the testing methodology and current results.