Evaluation of applicability of modified vector space representation for in-VM malicious activity detection in Cloud

Abstract

Malware writers use increasingly complex evasion mechanisms to ensure the concealment of malware against standard anti-malware suites. To identify malware through its behaviour, rather than its approach is an interesting venue of exploration. System call traces are highly indicative of a process behaviour. However, it is difficult to acquire system calls of all processes running on a physical machine. Fortunately, the same cannot be said for the virtual machines, owing to the advancement of Virtual Machine Introspection (VMI) techniques. This opens up the possibility of utilizing system call information for malicious activity detection. In this paper, we study different representations of system call information and evaluate their applicability for in- VM malicious activity detection in Cloud environment.