FProbe: Detecting Stealthy DGA-based Botnets by Group Activities Analysis

Nowadays, we have witnessed the rise of botnet malicious activities. These botnets, as expected, are launched by Domain generation algorithm (DGA) to evade detection. There is a growing concern that the artificially designed DGA detection features are being vulnerable to attackers, where any well-designed manipulations would evade these existing feature-based detection and even the more robust behavior-based detection. One common point of existing evasion for behavior detection is using domain names with low query rate. In this paper, we propose FProbe, a novel technology using co-occurrence matrix and relaxed clustering procedure, which performs excellent performance in the scene of detecting low query rate and multi-domain evasion. We use a simple intuition, that is, DGA queries have a strong correlation between temporal and spatial features, but these temporal and spatial correlations are not very synchronous. The FProbe uses the co-occurrence matrix, which is widely used in the field of product recommendation and word frequency co-occurrence, and use these unsupervised methods to cluster infected hosts. In particular, through this matrix, we can quickly and effectively locate infected hosts in the scene of low query rate, instead of discarding the domain for its high threshold. Then, we use the relax association rules of Frequent Sequence Tree to cluster related domain names, and use supervised learning to determine malicious clusters. The FProbe has been evaluated in the campus network (4000 active users in peak load hours) and ISP DNS traffic (one billion queries per hour). The experimental results ( 96.3% accuracy rate of 1.9% false positive on average) illustrate the efficiency and accuracy of FProbe.