POSTER: Static ROP Chain Detection Based on Hidden Markov Model Considering ROP Chain Integrity

Abstract

Return-oriented programming (ROP) has been crucial for attackers to evade the security mechanisms of operating systems. It is currently used in malicious documents that exploit viewer applications and cause malware infection. For inspecting a large number of commonly handled documents, high-performance and flexible-detection methods are required. However, current solutions are either time-consuming or less precise. In this paper, we propose a novel method for statically detecting ROP chains in malicious documents. Our method generates a hidden Markov model (HMM) of ROP chains as well as one of benign documents by learning known malicious and benign documents and libraries used for ROP gadgets. Detection is performed by calculating the likelihood ratio between malicious and benign HMMs. In addition, we reduce the number of false positives by ROP chain integrity checking, which confirms whether ROP gadgets link properly if they are executed. Experimental results showed that our method can detect ROP-based malicious documents with no false negatives and few false positives at high throughput.