A Novel Hybrid Mobile Malware Detection System Integrating Anomaly Detection With Misuse Detection

As the dominator of the Smartphone operating system market, Android has attracted the attention of malware authors and researchers alike. The number of Android malware is increasing rapidly regardless of the considerable number of proposed malware analysis systems. In this paper, by taking advantages of low false-positive rate of misuse detection and the ability of anomaly detection to detect zero-day malware, we propose a novel hybrid detection system based on a new open-source framework CuckooDroid, which enables the use of Cuckoo Sandbox's features to analyze Android malware through dynamic and static analysis. Our proposed system mainly consist of two parts, a misuse detector performing known malware detection and classification through combining static analysis with dynamic analysis; an anomaly detector performing abnormal apps detection through dynamic analysis. We evaluate our method with 5560 malware samples and 12000 benign samples. Experiments shows that our misuse detector with hybrid analysis can accurately detect and classify malware samples with an average positive rate 98.79%, 98.32% respectively; it is worth noting that our anomaly detector by dynamic analysis is capable of detecting zero-day malware with a low false negative rate (1.24%) and acceptable false positive rate (2.24%). Our proposed detection system is mainly designed for App store markets and the ordinary users who can access our system through mobile cloud service.