SMODIC: A Model Checker for Self-modifying Code

Abstract

In this paper, we present SMODIC, a model checker for self-modifying binary codes. SMODIC uses Self Modifying Pushdown Systems (SM-PDS) to model self-modifying binary code. This allows to faithfully represent the program’s stack as well as the self-modifying instructions of the program. SMODIC takes a self-modifying binary code or a self modifying pushdown system as input. It can then perform reachability analysis and LTL/CTL model-checking for these models. We successfully used SMODIC to model-check more than 900 self-modifying binary codes. In particular, we applied SMODIC for malware detection, since malwares usually use self-modifying instructions, and since malicious behaviors can be described by LTL or CTL formulas. In our experiments, SMODIC was able to detect 895 malwares and to prove that 200 benign programs were benign. SMODIC was also able to detect several malwares that well-known antiviruses such as Bit-Defender, Kinsoft, Avira, eScan, Kaspersky, Baidu, Avast, and Symantec failed to detect. SMODIC can be found in https://lipn.univ-paris13.fr/~touili/smodic