Program Obfuscation via ABI Debiasing

Abstract

The Itanium ABI is the most popular C++ ABI that defines data structures essential to implement underlying object-oriented concepts in C++. Specifically, name mangling rules, object and VTable layouts, alignment, etc. are all mandated by the ABI. Adherence to the ABI comes with undesirable side effects. While it allows interoperability, past research efforts have shown that it provides robust inference points that an attacker can leverage to reveal sensitive design information through binary reverse engineering. In this work, we aim to reduce the ability of an attacker to successfully reverse engineer a binary. We do this via removal of what we call ABI Bias, i.e., the reverse engineering bias that manifests due to a compiler’s adherence to the ABI. Specifically, we identify two types of ABI biases that are central to past reverse engineering works on C++ binaries: VTable ordering bias and Function Pointer bias. We present compiler-based techniques that can correctly and efficiently debias a given binary from the aforementioned biases. We evaluate our proof-of-concept implementation on a corpus of real world programs for binary size, correctness and performance. We report an average increase of 1.42% in binary size compared to the baseline, very low performance overhead and lastly, correct execution of evaluation programs in comparison to the baseline. Finally, we demonstrate efficacy of our approach by hindering DeClassifier, a state-of-the-art C++ reverse engineering framework.