Hybrid Analysis for Mining Network Protocol's Hidden Behavior

Abstract

Reverse unknown protocol's hidden behavior has played an important role in the field of network security. The proposed work takes the captured messages and the binary code that implement the protocol both as the studied object. Dynamic Taint Analysis combined with Static Analysis is used for protocol analyzing. Firstly, monitor and analyze the process of protocol program parses the message in the virtual platform HiddenDisc prototype system developed by ourselves, record the protocol's public behavior, then based on our proposed Hidden Behavior Perception and Mining algorithm, static analyze the protocol's hidden behavior trigger conditions and hidden behavior instruction sequences. According to the hidden behavior trigger conditions, new protocol messages with the sensitive information are generated, and the hidden behaviors are executed by dynamic triggering. HiddenDisc prototype system can sense, trigger and analyze the protocol's hidden behaviors. According to the statistical analysis results, we propose the evaluation method of Protocol Execution Security. The experimental results show that the present method can accurately mining the protocol's hidden behaviors, and can evaluate unknown protocol's execution security.