DEMO: top-k cardinality estimation with HyperLogLog sketches

Abstract

A recurring task in security monitoring consists in finding scan-type flows, namely flows which exhibit a large cardinality in terms of number of distinct source/destination addresses, or in most generality packet-level identifiers (e.g. ports, header fields, etc). But cardinality estimation requires to “remember” the identifiers seen in the past, and becomes quite challenging when the goal is to implement per-flow distinct count at wire speed, while maintaining high processing throughput and limited memory footprint. In this demo, we will show how to use HyperLogLog sketches to implement an efficient and innovative top-k cardinality estimation algorithm, called FlowFight. The algorithm has been tested and integrated in a full-fledged software router such as Vector Packet Processor.