Network Message Field Type Clustering for Reverse Engineering of Unknown Binary Protocols

Abstract

Reverse engineering of unknown network protocols based on recorded traffic traces enables security analyses and debugging of undocumented network services. One important step in protocol reverse engineering is to determine data types of message fields. Existing approaches for binary protocols (1) lack comprehensive methods to interpret message content and determine the data types of discovered segments in a message and (2) assume the availability of context, which prevents the analysis of complex and lower-layer protocols. Overcoming these limitations, we propose the first generic method to analyze message field data types in unknown binary protocols by clustering of segments with the same data type. Our extensive evaluation shows that our method in most cases provides clustering of up to 100 % precision at reasonable recall. Particularly relevant for use in fuzzing and misbehavior detection, we increase the coverage of message bytes over the state-of-the-art to 87 % by almost a factor of 30. We provide an open-source implementation to allow follow-up works.