Dynamic Self-modifying Code Detection Based on Backward Analysis

Abstract

Self-modifying code (SMC) is widely used in obfuscated program for enhancing the difficulty in reverse engineering. The typical mode of self-modifying code is restore-execute-hide, it drives program to conceal real behaviors at most of the time, and only under actual running will the real code be restored and executed. In order to locate the SMC and further recover the original logic of code for guiding program analysis, dynamic self-modifying code detecting method based on backward analysis is proposed. Our method first extracts execution trace such as instructions and status through dynamic analysis. Then we maintain a memory set to store the memory address of execution instructions, the memory set will update dynamically while backward searching the trace, and simultaneously will we check the memory write address to match with current memory set in order to identify the mode "modify then execute". By means of validating self-modifying code which is identified via above procedures, we can easily deobfuscate the program which use self-modifying code and achieve its original logic. A prototype that can be applied in self-modifying code detection is designed and implemented. The evaluation results show our method can trace the execution of program effectively, and can reduce the consumption in time and space.