Protocol reverse engineering through dynamic and static binary analysis

Abstract

This paper presents a new method for protocol reverse engineering, which combines both the dynamic and static binary analysis. Our work not only does precise positioning on the field and its length, but also gives the field attributes accurately. According to different instructions and the current program structure, we can infer the message format validly. To prove the method is sound and effective, we build a prototype tool – NetProtocolFinder, and select some documented protocol and undocumented protocol messages as the test instances respectively. Results of our experiments show that the tool can not only extract the message format from protocols effectively, but also speculate the state machine model through relevant field attributes conveniently.