Stochastic Optimization of Program Obfuscation

Abstract

Program obfuscation is a common practice in software development to obscure source code or binary code, in order to prevent humans from understanding the purpose or logic of software. It protects intellectual property and deters malicious attacks. While tremendous efforts have been devoted to the development of various obfuscation techniques, we have relatively little knowledge on how to most effectively use them together. The biggest challenge lies in identifying the most effective combination of obfuscation techniques. This paper presents a unified framework to optimize program obfuscation. Given an input program P and a set T of obfuscation transformations, our technique can automatically identify a sequence seq = ‹t1, t2, …, tn› (∀i ∈ [1, n]. ti ∈ T), such that applying ti in order on P yields the optimal obfuscation performance. We model the process of searching for seq as a mathematical optimization problem. The key technical contributions of this paper are: (1) an obscurity language model to assess obfuscation effectiveness/optimality, and (2) a guided stochastic algorithm based on Markov chain Monte Carlo methods to search for the optimal solution seq. We have realized the framework in a tool Closure* for JavaScript, and evaluated it on 25 most starred JavaScript projects on GitHub (19K lines of code). Our machinery study shows that Closure* outperforms the well-known Google Closure Compiler by defending 26% of the attacks initiated by JSNice. Our human study also reveals that Closure* is practical and can reduce the human attack success rate by 30%.