q-MAX: A Unified Scheme for Improving Network Measurement Throughput

Network measurement is an essential building block for a variety of network applications such as traffic engineering, quality of service, load-balancing and intrusion detection. Maintaining a per-flow state is often impractical due to the large number of flows, and thus modern systems use complex data structures that are updated with each incoming packet. Therefore, designing measurement applications that operate at line speed is a significant challenge in this domain. In this work, we address this challenge by providing a unified mechanism that improves the update time of a variety of network algorithms. We do so by identifying, studying, and optimizing a common algorithmic pattern that we call q-MAX. The goal is to maintain the largest q values in a stream of packets. We formally analyze the problem and introduce interval and sliding window algorithms that have a worst-case constant update time. We show that our algorithms perform up to X20 faster than library algorithms, and using these new algorithms for several popular measurement applications yields a throughput improvement of up to X12 on real network traces. Finally, we implemented the scheme within Open vSwitch, a state of the art virtual switch. We show that q-MAX based monitoring runs in line speed while current monitoring techniques are significantly slower.