Large-scale machine learning-based malware detection: confronting the "10-fold cross validation" scheme with reality

To address the issue of malware detection, researchers have recently started to investigate the capabilities of machine-learning techniques for proposing effective approaches. Several promising results were recorded in the literature, many approaches being assessed with the common "10-Fold cross validation" scheme. This paper revisits the purpose of malware detection to discuss the adequacy of the "10-Fold" scheme for validating techniques that may not perform well in reality. To this end, we have devised several Machine Learning classifiers that rely on a novel set of features built from applications' CFGs. We use a sizeable dataset of over 50,000 Android applications collected from sources where state-of-the art approaches have selected their data. We show that our approach outperforms existing machine learning-based approaches. However, this high performance on usual-size datasets does not translate in high performance in the wild.