RandHeap: Heap Randomization for Mitigating Heap Spray Attacks in Virtual Machines

Abstract

Virtual machines are an integral component of our present software systems infrastructure, including the web, and are here to stay. Web browsers like Google Chrome and Mozilla Firefox uses virtual machines to execute JavaScript code. Java Virtual Machines (JVMs) use just-in-time compilers to compile Java byte code to machine code. However, with the increasing use of virtual machines, they are also susceptible to security attacks. One such class of attack is the heap spray attack, wherein the attacker populates the heap with malicious code and exploits a vulnerability to jump to the populated malicious code in the heap, thereby enabling arbitrary code execution. In this paper, we present RandHeap, a technique to randomize the heap layout to detect and prevent heap spray attacks. RandHeap randomizes the heap in three different ways: (i) by randomizing object layout, (ii) by randomizing array layout, and (iii) by encrypting data stored on the heap. Using RandHeap, we were able to detect and prevent several heap spray attacks. For the evaluation of RandHeap, we implemented the concept of RandHeap in Google V8 and JikesRVM. We executed Octane 2.0 Benchmarks on Google V8 and Dacapo 9.12 Benchmarks on JikesRVM. Observations show that heap randomization using RandHeap is accompanied with low overhead and modest memory requirement. We implemented heap spraying attacks in Google V8 and JikesRVM and found that RandHeap was able to detect and prevent the attacks successfully.