Network security via reverse engineering of TCP code: vulnerability analysis and proposed solutions

Abstract

The transmission control protocol/Internet protocol (TCP/IP) suite is widely used to interconnect computing facilities in modern network environments. However, there exist several security vulnerabilities in the TCP specification and additional weaknesses in a number of its implementations. These vulnerabilities may enable an intruder to "attack" TCP-based systems, allowing him/her to "hijack" a TCP connection or cause denial of service to legitimate users. We analyze TCP code via a "reverse engineering" technique called "slicing" to identify several of these vulnerabilities, especially those that are related to the TCP state-transition diagram. We discuss many of the flaws present in the TCP implementation of many widely used operating systems, such as SUNOS 4.1.3, SVR4, and ULTRIX 4.3. We describe the corresponding TCP attack "signatures" (including the well-known 1994 Christmas Day Mitnick Attack) and provide recommendations to improve the security state of a TCP-based system, e.g., incorporation of a "timer escape route" from every TCP state.