Exploring the state space of an application protocol: A case study of SMTP

In this work, we explore the state space of a network application protocol by employing genetic programming techniques. To this end, we target Simple Mail Transfer Protocol (SMTP), which is a well-known and open protocol on the Internet. In order to achieve our goal, we aim to evolve the payload such that solution individuals result in an email being sent successfully through the targeted server. The proposed system implements an archive paradigm where, upon completion of the evolutionary process, a collection (archive) of solutions are presented. Specifically, they can all achieve the goal, but each does so in a unique manner. This collection allows us to examine the state space of the application protocol, giving us the ability to verify that these variations are either intended by the protocol, or should be addressed for security reasons.