Olivier Langlois's blog

I have submitted a patch for the bugzilla bug 3606:

http://bugzilla.dre.vanderbilt.edu/show_bug.cgi?id=3606

Fix function ACE::handle_ready + SOCK_Test modif to test the fix

The patch include the following:

1. Fix for the bug 3606 in function ACE::handle_ready
2. Modif of the unittest SOCK_Test to validate the fix
3. Removal of the define ACE_HAS_LIMITED_SELECT

There has been a very long thread about it last summer on the ACE mailing list and I was arguing that it was dangerous to use select() when poll() is available because if you pass an handle whose value that is higher than FD_SETSIZE (typically 1024), it causes a buffer overflow on the stack which is hard to debug.

At the end, a consensus emerged that ACE_HAS_LIMITED_SELECT should be removed. Actually if some platform would benefit from that, it would be safer to define ACE_HAS_UNLIMITED_SELECT instead.

4. Remove the duplicated code from ACE::handle_ready() at different locations to replace it with the appropriate ACE namespace function call.

Upon reception of RST segment, the receiving side will immediately abort the connection. This statement has more implications than just meaning that you will not be able to receive or send any more data to/from this connection. It also implies that any unread data still in the TCP reception buffer will be lost. This information can be found in TCP/IP Internetworking volume 2 and Unix Network Programming Volume 1 third edition but in my opinion those books do not put enough emphasis on that detail and if you are not reading these books to find this exact detail, it might slip away from your attention.

Now, what are the conditions to receive a RST segment? The easiest way is to enable the SO_LINGER option with a timeout value of 0 on a server socket. As soon as the connection will be closed by the server, it will send a RST. You can learn more about SO_LINGER in the book Unix Network Programming Volume 1 third edition. The other possibility is if a server receives data after having closed the connection. If you enable the LINGER option, sending RST is the expected behavior but you can easily be bitten by the second possibility and here is an example on how it can happen.

Imagine a HTTP server that limits the number of simultaneous connections. Upon accepting a new connection, it might compare a global connection counter against a configured limit and if the counter has reached the limit then immediately send back a 503 error and close the connection. Do you see the problem? The server is closing the connection before receiving the HTTP request. There is a high probability that the client will never receive the 503 reply because a RST segment will immediately follow the reply.

A not too good solution would be to call shutdown(SHUT_WR) before closing the socket. What this will do is force the server to send a FIN segment before the RST segment. I have tried this solution and this seems to greatly improve the probability that the client application will receive the data before receiving the RST segment.

A better way is to program the server to read the request even if it has no intention to actually process it. This ensures that no RST will be sent unintentionally.

There remains one question. What exactly means data reception on a closed connection? Does that only include new received TCP segment or could data still in the TCP receive buffer of a closing connection be considered as data arrived after the connection has been closed? My intuition is that this question is open to interpretation and the TCP behavior will vary from one platform to the other. However, even if your platform does not react with a RST if a TCP connection receive buffer is not empty when it is closing it; my opinion is that you should not rely on that since this is a race condition. You could be closing the connection before or after receiving the client request depending on the RTT (Round Time Trip) of the connection.

In the book Internetworking with TCP/IP volume 2 - Design, implementation, and internals at the chapter 13 (TCP: Output processing), section 13.16 Other TCP Procedures, the source code of the function tcpiss() called to obtain the Initial Send Sequence (ISS) is presented. The function works by initializing a static variable with a clock counter value and then increment that value to TCPINCR whose value is 904.

No explanation is provided about the 904 value and there is even an exercise asking the reader to contact the coauthor David Stevens by e-mail to know why they are using 904. I first tried to find the answer on the net before contacting them but the information was unavailable.

So here is the answer of Douglas E. Comer, coauthor of the book, on the topic for the next fellow looking for the same answer:

It's an inside joke. The TCP standard says to choose a value other than 1.
My co-author, Dave Stevens, and I tossed a coin, and we used his birthday (September 4th). There are a couple of other fun items in the book (hint: look in the index).

Regards,
Doug Comer

After reading Mr. Comer reply, I have checked the index and there is a list of well-known constants such as the world renowned 1.3.6.1.2.1 !

While we were using an Apache ActiveMQ broker at my job. It has been reported by the operators last week that the broker could not open new sockets because it has reached the maximum limit of open file descriptors. The operators have also mentioned that the majority of the TCP sockets where in the CLOSE_WAIT state. My initial reaction has been to confuse the TCP state CLOSE_WAIT with the TIME_WAIT state.

Yesterday, someone else reported the same problem on the Apache ActiveMQ developers’ mailing list and I have noted by reading posts from people trying to suggest workarounds that I was not the only one to mix up the TCP state CLOSE_WAIT and TIME_WAIT. So this gave me the idea to take this opportunity to write about the differences between these 2 states since apparently there is a widespread need for clarifications on that matter.

A TCP connection goes into the CLOSE_WAIT state when it receives a FIN segment from its peer. From that point the connection becomes half-duplex and the TCP connection will not receive any new data from its peer but it will still be able to send any amount of data back to the peer. Hence, the socket will stay there as long as the server does not call close() explicitly on the socket.

I am not sure on this but I think this is exactly what the state name means. The socket is waiting for the application to call close() before going away.

TIME_WAIT is the TCP state a connection will go when it performs an active close (it is initiating the connection shutdown) after having received the peer FIN segment and sent it back an acknowledgment for the FIN.

The connection will stay in that state for 2 times the maximum segment live (MSL). As an interesting side note, the MSL has nothing to do with IP time to live (TTL) or the TCP connection round time trip (RTT). The MSL value is a constant and varies from one TCP implementation to another. RFC 793 suggests using 2 minutes for the MSL and the implementation shown in this book is using 2 minutes but 4.4BSD is using 75 seconds and Linux close to 4 minutes. The reason why MSL is unrelated to IP TTL and RTT is that a TCP segment can outlive the IP packet carrying it even if routers takes longer than usual to deliver it since a TCP stack will retransmit it.

TIME_WAIT is needed because it is possible that the TCP acknowledgment for the received FIN gets lost (or any segment containing data prior the FIN). In that case, the peer will try to retransmit its FIN for the MSL period. Waiting for a longer period than MSL is the only choice TCP can make as there are no acknowledgements for ACKs (that would be recursive anyway and solve nothing if ACKs were acknowledged).

I have been wondering for some time why TCP does not wait 2 times MSL before leaving the LAST_ACK state like it does for TIME_WAIT since there are scenarios where TCP could receive packets from the previous connection after leaving LAST_ACK and I think that I have understand why. In TIME_WAIT, you want to handle peer retransmission otherwise it could think that its last segments have never been received. In LAST_ACK, there is nothing else to acknowledge so it does not matter if TCP reset the connection after leaving LAST_ACK.

For instance, let’s say that the stack retransmits its FIN and both the original FIN and the retransmitted one get acknowledged. What will happen is that at the reception of the first ACK, TCP will release its Transmission Control Block (TCB) associated to this connection and at the reception of the second ACK, it will reply with a RST. It is no big deal. The other peer will be in the TIME_WAIT state and will just conclude that the peer is gone when it will read the RST.

A NAT router is a device that allow many computers to share the same IP address. You can learn more about NAT itself from Internetworking with TCP/IP, Vol 1 (5th Edition). One drawback of using NAT is that to communicate with someone outside the private network, you must initiate the communication from the private network. This is all fine for contacting a web server but this is making peer to peer applications such as file sharing or gaming harder. There are some techniques to perform NAT traversal but what makes the matter worse is that NAT routers have a different NAT behavior from a model to the other and as if it was not bad enough, these behaviors are totally undocumented. Things are about to change as the RFC 4787 describes a set of behaviors that a router should have to be gaming friendly. So my advice as a person who has worked at Quazal, a big well known multiplayer online middleware company, if you want the best NAT router for online gaming for your video game console (Xbox, Xbox360, Nintendo Wii, Playstation 2, Playstation 3 or PSP), you should look for a router that is RFC 4787 compliant. They might be hard to find now as the RFC has been released just few months ago but if I personnally was looking to purchase a new router, that is what I would be looking for.

The basic principle to perform NAT traversal is that you need a server on the public network that will be used as a traversal probe relay. Lets say that client A and client B that are located behind 2 distinct NAT routers. They first have to contact the server. The server must keep the address from which it receive data from the clients. These addresses are the public address of the clients behind the NAT router as seen from the server. When client A wants to establish a connection with client B, it will first request the client B public address from the server and start sending probes to B public address. At the same time, client A will send a request to B by using the server as a relay to start sending probes to itself. What will happen is since both clients try to establish a connection with each other at the same time, there should be a midair collision where a hole has been created for the other on each of their respective router. This technique is called hole punching and there is an RFC describing a protocol that implement this principle. The protocol name is STUN (Simple Traversal of UDP for NAT).

NAT traversal for TCP is much harder because TCP handshaking for establishing a connection is asymmetric. To workaround that problem, you have to trick the TCP protocol and the router that might monitor the TCP handshake. There is not yet an official protocol but some proposals exist. One of them is STUNT (STUN for TCP) but the name is funny because it has also a second meaning. The way STUNT works is that both side will initiate a connection by sending a SYN packet with a short TTL (Time To Live). The TTL value must be carefully choosen so the packet goes out of the private network but does not reach the destination. Then the stack itself or a packet sniffer must read the sequence number contained in the SYN message and send the read value to the STUNT server. The server will then impersonate the other endpoint and spoof the SYN/ACK reply. Then the last step of the connection which is to send the final ACK can be sent as usual and if everything has worked fine, the TCP connection will be established normally. I will stop the NAT traversal explanation here as there are a lot of good resources on the net. Here are the most interesting:

• RFC 2663
• RFC 3022
• RFC 3027
• RFC 3489
• RFC 4787
• http://en.wikipedia.org/wiki/STUN
• http://en.wikipedia.org/wiki/UDP_hole_punching
• http://nutss.gforge.cis.cornell.edu/publications.php