Adaptive header identification and unsupervised clustering strategy for enhanced protocol reverse engineering

Abstract

Protocol reverse engineering is critical for ensuring network security and understanding proprietary communication mechanisms. Most traditional network trace-based methods face challenges such as high computational complexity, excessive memory usage, and sensitivity to payload variations. In this paper, we propose a method that integrates adaptive message header recognition with unsupervised clustering strategies for protocol reverse engineering. Utilizing mean entropy change and change point detection algorithms, our method automatically identifies message headers, reducing the impact of payload variations on similarity measurements. Following this, our method significantly reduces computational resource consumption while maintaining clustering performance, by clustering based on a small set of selected core samples of message headers and assigning the remaining samples to existing categories. Moreover, leveraging the identified message headers, we incorporate a hierarchical format inference technique and design a function code field detector, which enhances the accuracy and efficiency of protocol reverse engineering. Our evaluation across eight widely used protocols demonstrates that our method achieves homogeneity and completeness scores of 0.94 and 0.74, respectively, in message type identification. These results significantly outperform existing protocol reverse engineering tools on the same datasets: MFD&DBSCAN (0.31, 0.73), NEMETYL (0.73, 0.64), and Netzob (0.34, 0.76). Furthermore, our method achieves a perfection score $1.2\times$ higher than Binaryinferno in format inference.