A Fragment Hashing Approach for Scalable and Cloud-Aware Network File Detection

Monitoring networks for the presence of some particular set of files can, for example, be important in order to avoid exfiltration of sensitive data, or combat the spread of Child Sexual Abuse (CSA) material. This work presents a scalable system for large-scale file detection in high-speed networks. A multi-level approach using packet sampling with rolling and block hashing is introduced. We show that such approach together with a well tuned implementation can perform detection of a large number of files on the network at 10 Gbps using standard hardware. The use of packet sampling enables easy distribution of the monitoring processing functionality, and allows for flexible scaling in a cloud environment. Performance experiments on the most run-time critical hashing parts shows a single-thread performance consistent with 10Gbps line rate monitoring. The file detectability is examined for three data sets over a range of packet sampling rates. A conservative sampling rate of 0.1 is demonstrated to perform well for all tested data sets. It is also shown that knowledge of the file size distribution can be exploited to allow lower sampling rates to be configured for two of the data sets, which in turn results in lower resource usage.