In the classical TCP, the opening phase of TCP connection establishment uses a three-way handshake to negotiate the initial sequence number for each direction. In the Secure TCP, this phase also handles negotiations of the method of TCP segment encryption and encryption key exchanges. Therefore, in the Secure TCP, the three-way handshake is extended to handle the negotiation.
The three-way handshake is a connection establishment protocol. The setup procedure of the protocol when entity A initiate shown in Figure 1. First, A sends a SYN segment to B in order to check that B gets ready for establishing TCP connection. Second, when B receives the SYN segment A sent and is ready to start the TCP session, it sends a SYN and ACK segment back to A. This ACK advertises an arrival of the first SYN segment to A. Finally, A sends ACK segment for the second SYN and ACK segment B sent.
Figure 1: Three-way handshake of TCP
In the Secure TCP, this three-way handshake is extended in order to deliver negotiation data and key exchange data. The procedure of the extended protocol is shown in Figure2. First, A sends a SYN segment with the negotiation data (NEGO) to B. The negotiation data is a list of encryption methods available at A used for segment protection. In other words, A advertises its encryption capability to B using this negotiation phase. If B receives the SYN + NEGOa segment from A, then B checks its own encryption capability and picks one of the methods advertised by A in the segment. The method B picks (NEGOb) will be used for TCP segment encryption in the further segment exchanges. Then, B returns SYN+ACK segment with a public key(KEYp) for encryption. This public key will be used for encryption of session keys applied to the stream encryption. Next, A sends a session keys (KEYs) for the stream encryption to B. The session keys will be applied to streams for data encryption. Finally, B sends back ACK segment for the ACK + KEYs segment A sent.
Figure 2: Extension of Three-way handshake