Scalable and Fast Hierarchical Clustering of IoT Malware Using Active Data Selection

Abstract

The number of IoT malware specimens has in-creased rapidly and diversified in recent years. To efficiently analyze a large number of malware specimens, we aim to reduce the calculation cost by clustering specimens with an incomplete distance matrix. Towards this goal, we applied the active clustering algorithm. In this algorithm, Mean-Field An-nealing (MFA) is used to determine the best clustering and the expected value of information criterion to actively choose which pair of specimens to observe its distance. We evaluated the active clustering algorithm with 3,008 mal ware specimens. By applying the active clustering algorithm, we only need to calculate 2.6 % of the whole distance matrix. The active clustering algorithm achieved 86.9% of family name accuracy and 96.5% of architecture name accuracy. Furthermore, the active clustering algorithm achieved the same level of accuracy as our former clustering algorithm with only 2.6 % observation, while our former algorithm needs to observe 7.2 % of the distance matrix. The observation reduction rate is 64 %.