Fast payload-based flow estimation for traffic monitoring and network security

Real-time IP flow estimation has many potential applications in network management, monitoring, security, and traffic engineering. Existing techniques typically rely on flow definitions being constrained as subsets of the fields in packet headers. This makes flow-membership tests relatively inexpensive. In this paper, we consider a more general flow estimation problem that needs complex packet-payload based tests for flow-membership. An example is to estimate traffic with common strings in the payload and detect potential virus signatures for early alarm generation. We develop a fast, memory efficient algorithm for solving this problem as a variant of the longest common subsequence problem. This is done via an application of Rabin fingerprinting in combination with bloom filters. Both analysis and simulation show the effectiveness of the developed method.