HoneyAgent: Detecting malicious Java applets by using dynamic analysis

Abstract

Malicious Java applets are widely used to deliver malicious software to remote systems. In this work, we present HoneyAgent which allows for the dynamic analysis of Java applets, bypassing common obfuscation techniques. This enables security researchers to quickly comprehend the functionality of an examined applet and to unveil malicious behavior. In order to trace the behavior of a sample as far as possible, HoneyAgent is further able to simulate various vulnerabilities allowing analysts for example to identify the malware that should finally be installed by the applet. In our evaluation, we show that HoneyAgent is able to reliably detect malicious applets used by common exploit kits with no false positives. By using a combination of heuristics as well as signatures applied to observed method invocations, HoneyAgent is further able to identify exploited common vulnerabilities and exposures in many cases.