Lightweight Examination of DLL Environments in Virtual Machines to Detect Malware

Abstract

Since it becomes increasingly difficult to trick end users to install and run executable files from unknown sources, attackers refer to stealthy ways such as manipulation of DLL (Dynamic Link Library) files to compromise user computers. In this paper, we propose to develop mechanisms that allow the hypervisor to conduct lightweight examination of DLL files and their running environment in guest virtual machines. Different from the approaches that focus on static analysis of the DLL API calling graphs, our mechanisms conduct continuous examination of their running states. In this way, malicious manipulations to DLL files that happen after they are loaded into memory can also be detected. In order to maintain non-intrusive monitoring and reduce the impacts on VM performance, we avoid examinations of the complete DLL file contents but focus on the parameters such as the relative virtual addresses (RVA) of the functions. We have implemented our approach in Xen and conducted experiments with more than 100 malware of different types. The experiment results show that our approach can effectively detect the malware with very low increases in overhead at guest VMs.