Olivier Langlois's blog

I saw a very entertaining youtube video where the host has tested 10 Gbits Ethernet cards on a LAN to perform file transfers. He somehow could not reach the maximum theoritical transfer speed. Here is my explanation why and it is probably that simple:

I do not know what was the protocol underneath for the transfer but if it is TCP, there is something called the bandwidth-delay product that says that in order to keep the network pipe full, you need sufficient TCP send and receive buffers.

Let me show you how to compute that. RTT is basically given by ping. Lets assume worse case scenario with a RTT of 0.2 mSec. 10Gbit/sec is 1250MB/sec or 1,250,000,000 B/sec.

1,250,000,000 B/sec * 0.0002 sec = 250,000B or 250KB. Last time I have checked (I'm a Linux guy) default Windows TCP buffers size were something around 16KB or perhaps 64KB. These are the parameters that you would need to play with to get close to the 10GB/sec rates.

You can change the default value through the registry or on a per connection basis with the socket API.

If we take your maximum transfer rate in your video which was roughly about 360MB/sec and plug it in the bandwidth delay product formula, I get 72KB which is very close to 64KB. Pretty sure that this is your problem!

It is an ok Gigabit router with its wireless functionality working fine. However when connecting a PS3 on the router and let it test the router capabilities, the router fails the UPnP testing. In my opinion, this is inexcusable when you sell a product at premium price and you market it as a "gaming" router, you would expect the company to have tested it with all the mainstream consoles.

I have contacted Cisco/Linksys customer support chat service to report the problem and get assistance and basically, I have been told something like that the problem was my fault and that I had to change some obscure and unrelated WiFi settings to fix my problem.

I was not satisfied by the answer and I decided to see for myself why the PS3 is reporting a UPnP failure with that router.

2 tools have been required for the analysis.

1. Wireshark
2. An Ethernet hub

A hub is a piece of networking hardware allowing you to share an Ethernet link between more than 2 network devices. Those were common in the 90s when processing power and memory were expensive. Today they have been replaced by the more robust and powerful Ethernet switches. My setup has been to plug my PC between the Linksys router and my PS3 and let Wireshark running on my PC sniff the UPnP exchange between them.

Before going in the details of the PS3 UPnP test, I want to share some interesting details about the whole PS3 Internet testing which is close to be undocumented on the Internet

Testing Internet Connection:

It does it by doing a HTTP request on fus01.ps3.update.playstation.net.


GET /update/ps3/list/us/ps3-updatelist.txt HTTP/1.1
Host: fus01.ps3.update.playstation.net
Connection: Keep-Alive
Accept-Encoding: identity
User-Agent: PS3Update-agent/1.0.0 libhttp/1.0.0

HTTP/1.1 200 OK
Server: Apache
ETag: "4d8c4dbf774c6349ad778577e53bd8c7:1285037260"
Last-Modified: Tue, 21 Sep 2010 02:47:40 GMT
Accept-Ranges: bytes
Content-Length: 252
Content-Type: text/plain
Date: Mon, 06 Dec 2010 03:22:33 GMT
Connection: keep-alive

# US
Dest=84;CompatibleSystemSoftwareVersion=3.5000-;
Dest=84;ImageVersion=0000b437;SystemSoftwareVersion=3.5000;CDN=http://dus01.ps3.update.playstation.net/update/ps3/image/us/2010_0921_0215e26d1dadeb950471a9c3397a140a/PS3UPDAT.PUP;CDN_Timeout=30;


Test PSN connection:

It does so by establishing an HTTPS connection with auth.np.ac.playstation.net. Obviously since it is encrypted, I cannot comment much about it.

NAT Type testing.

The PS3 sends a series of STUN (Session Traversal Utilities for NAT) (I am mentionning STUN in another blog entry for the curious) requests to us.np.stun.playstation.net

So now, lets get back to the UPnP testing. Here is a short overview of the whole procedure:

• Broadcast a request to find all network devices that are a InternetGatewayDevice
• Query its capability by requesting a XML file
• If the service WANIPConnection is supported, continue the test
• Call GetExternalIPAddress to obtain the router public Internet IP address
• Call AddPortMapping to open a public port on which all inbound traffic will be forwarded to the PS3

Here is how goes the exchange with the WRT330N router:


PS3 Broadcast:

M-SEARCH * HTTP/1.1
HOST: 239.255.255.250:1900
MAN: "ssdp:discover"
MX: 5
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1

2 seconds timeout. Retransmit the same request

After about 1.5 seconds after the 2nd retransmission, the router finally reply:

HTTP/1.1 200 OK
EXT:
SERVER: ipOS/7.2, UPnP/1.0, ipSSDPDevice/1.0
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
LOCATION: http://192.168.1.1/root.sxml
CACHE-CONTROL: max-age=1800
USN: uuid:1C7AE0B4-AF9D-3FD9-AC27-04FAE3357DD5::urn:schemas-upnp-org:device:InternetGatewayDevice:1
Content-Length: 0

PS3 request this:

GET /root.sxml HTTP/1.1
HOST: 192.168.1.1:80

PS3 UPnP request:

POST /wipconn HTTP/1.1
HOST: 192.168.1.1:4444
Content-Length: 290
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#GetExternalIPAddress"

Linksys Router:

HTTP/1.1 200 OK
SERVER: ipOS/6.8 UPnP/1.0 IGD/1.0
EXT:
Transfer-Encoding: Chunked

197
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><soap:Body><u:GetExternalIPAddressResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"><NewExternalIPAddress>xxx.xxx.xxx.xxx</NewExternalIPAddress></u:GetExternalIPAddressResponse></soap:Body></soap:Envelope>
0

2nd PS3 UPnP request:

POST /wipconn HTTP/1.1
HOST: 192.168.1.1:4444
Content-Length: 644
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping"

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:AddPortMapping xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
<NewRemoteHost></NewRemoteHost>
<NewExternalPort>3658</NewExternalPort>
<NewProtocol>UDP</NewProtocol>
<NewInternalPort>3658</NewInternalPort>
<NewInternalClient>192.168.1.109</NewInternalClient>
<NewEnabled>1</NewEnabled>
<NewPortMappingDescription>192.168.1.109:3658 to 3658 (UDP)</NewPortMappingDescription>
<NewLeaseDuration>0</NewLeaseDuration>
</u:AddPortMapping>
</s:Body>
</s:Envelope>

Linksys Router:

HTTP/1.1 200 OK
SERVER: ipOS/6.8 UPnP/1.0 IGD/1.0
EXT:
Transfer-Encoding: Chunked
151
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><soap:Body><u:AddPortMappingResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"></u:AddPortMappingResponse></soap:Body></soap:Envelope>
0


By looking the exchange it looks all good and in the router log, there is even an entry that shows that the Port mapping occured:


[INFO] Sun Dec 05 20:40:09 2010 UPnP renew entry 255.255.255.255 <-> 24.37.208.168:3658 <-> 192.168.1.109:3658 UDP timeout:-1 '192.168.1.109:3658 to 3658 (UDP)'


I initially believed that the error is the returned length of the router reply for the AddPortMapping request. It reports a length of 151 bytes but if you count them, I come to something around 340 chars! The GetExternalIPAddress call has the same problem. The reported size is 197 while in fact, I count around 410 chars!

However, a coworker of mine pointed out to me that the chunk size values are in hex as described in the RFC 2616. So I can only conclude that the problem comes from the PS3 that does not handle correctly HTTP chunked transfer coding in the context of UPnP exchange.

Another indication that the chunked transfer encoding in the replies is the culprit of the problem, it is that at the end of both exchange between the PS3 and the WRT330N is that the PS3 is sending back TCP RST segments to the router which means that the PS3 is closing its connection with the router before the router having finished to send its replies.

For your reference, here is an exchange with my new NetGear router, the WNDR37AV that works like a charm!


In less than 2 ms the reply to the M-SEARCH query comes back.

HTTP/1.1 200 OK
Cache-Control: max-age=1800
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
USN: uuid:12345678-0000-0000-0000-00000000abcd::urn:schemas-upnp-org:device:InternetGatewayDevice:1
EXT:
Server: Linux/2.6.15-1.2054_FC5 UPnP/1.0 miniupnpd/1.0
Location: http://192.168.1.1:5555/rootDesc.xml

PS3 request this:

GET /rootDesc.xml HTTP/1.1
HOST: 192.168.1.1:5555

PS3 UPnP request:

POST /ctl/IPConn HTTP/1.1
HOST: 192.168.1.1:5555
Content-Length: 290
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#GetExternalIPAddress"

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:GetExternalIPAddress xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
</u:GetExternalIPAddress>
</s:Body>
</s:Envelope>

Netgear reply:

HTTP/1.1 200 OK
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 356
Server: Linux/2.6.15-1.2054_FC5 UPnP/1.0 miniupnpd/1.0
Ext:

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><s:Body><u:GetExternalIPAddressResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"><NewExternalIPAddress>xxx.xxx.xxx.xxx</NewExternalIPAddress></u:GetExternalIPAddressResponse></s:Body></s:Envelope>

2nd PS3 UPnP request:
POST /ctl/IPConn HTTP/1.1
HOST: 192.168.1.1:5555
Content-Length: 644
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping"

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:AddPortMapping xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
<NewRemoteHost></NewRemoteHost>
<NewExternalPort>3658</NewExternalPort>
<NewProtocol>UDP</NewProtocol>
<NewInternalPort>3658</NewInternalPort>
<NewInternalClient>192.168.1.109</NewInternalClient>
<NewEnabled>1</NewEnabled>
<NewPortMappingDescription>192.168.1.109:3658 to 3658 (UDP)</NewPortMappingDescription>
<NewLeaseDuration>0</NewLeaseDuration>
</u:AddPortMapping>
</s:Body>
</s:Envelope>

Netgear reply:

HTTP/1.1 200 OK
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 260
Server: Linux/2.6.15-1.2054_FC5 UPnP/1.0 miniupnpd/1.0
Ext:

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><s:Body><u:AddPortMappingResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"/>
</s:Body>
</s:Envelope>


In conclusion, I am disapointed by the quality of the Linksys product and the lack of attention of details that the company did put in the QA before the release of that product since the UPnP bug could have been caught by a very simple test in less than 5 minutes. Apparently the bug is in the PS3 that does not support UPnP interaction with HTTP replies using chunked transfer coding but I was expecting to purchase a router compatible with my PS3 when I bought a gaming router. For that reason, I prefer the NetGear router.

Happy networking!

I have submitted a patch for the bugzilla bug 3606:

http://bugzilla.dre.vanderbilt.edu/show_bug.cgi?id=3606

Fix function ACE::handle_ready + SOCK_Test modif to test the fix

The patch include the following:

1. Fix for the bug 3606 in function ACE::handle_ready
2. Modif of the unittest SOCK_Test to validate the fix
3. Removal of the define ACE_HAS_LIMITED_SELECT

There has been a very long thread about it last summer on the ACE mailing list and I was arguing that it was dangerous to use select() when poll() is available because if you pass an handle whose value that is higher than FD_SETSIZE (typically 1024), it causes a buffer overflow on the stack which is hard to debug.

At the end, a consensus emerged that ACE_HAS_LIMITED_SELECT should be removed. Actually if some platform would benefit from that, it would be safer to define ACE_HAS_UNLIMITED_SELECT instead.

4. Remove the duplicated code from ACE::handle_ready() at different locations to replace it with the appropriate ACE namespace function call.

Upon reception of RST segment, the receiving side will immediately abort the connection. This statement has more implications than just meaning that you will not be able to receive or send any more data to/from this connection. It also implies that any unread data still in the TCP reception buffer will be lost. This information can be found in TCP/IP Internetworking volume 2 and Unix Network Programming Volume 1 third edition but in my opinion those books do not put enough emphasis on that detail and if you are not reading these books to find this exact detail, it might slip away from your attention.

Now, what are the conditions to receive a RST segment? The easiest way is to enable the SO_LINGER option with a timeout value of 0 on a server socket. As soon as the connection will be closed by the server, it will send a RST. You can learn more about SO_LINGER in the book Unix Network Programming Volume 1 third edition. The other possibility is if a server receives data after having closed the connection. If you enable the LINGER option, sending RST is the expected behavior but you can easily be bitten by the second possibility and here is an example on how it can happen.

Imagine a HTTP server that limits the number of simultaneous connections. Upon accepting a new connection, it might compare a global connection counter against a configured limit and if the counter has reached the limit then immediately send back a 503 error and close the connection. Do you see the problem? The server is closing the connection before receiving the HTTP request. There is a high probability that the client will never receive the 503 reply because a RST segment will immediately follow the reply.

A not too good solution would be to call shutdown(SHUT_WR) before closing the socket. What this will do is force the server to send a FIN segment before the RST segment. I have tried this solution and this seems to greatly improve the probability that the client application will receive the data before receiving the RST segment.

A better way is to program the server to read the request even if it has no intention to actually process it. This ensures that no RST will be sent unintentionally.

There remains one question. What exactly means data reception on a closed connection? Does that only include new received TCP segment or could data still in the TCP receive buffer of a closing connection be considered as data arrived after the connection has been closed? My intuition is that this question is open to interpretation and the TCP behavior will vary from one platform to the other. However, even if your platform does not react with a RST if a TCP connection receive buffer is not empty when it is closing it; my opinion is that you should not rely on that since this is a race condition. You could be closing the connection before or after receiving the client request depending on the RTT (Round Time Trip) of the connection.

In the book Internetworking with TCP/IP volume 2 - Design, implementation, and internals at the chapter 13 (TCP: Output processing), section 13.16 Other TCP Procedures, the source code of the function tcpiss() called to obtain the Initial Send Sequence (ISS) is presented. The function works by initializing a static variable with a clock counter value and then increment that value to TCPINCR whose value is 904.

No explanation is provided about the 904 value and there is even an exercise asking the reader to contact the coauthor David Stevens by e-mail to know why they are using 904. I first tried to find the answer on the net before contacting them but the information was unavailable.

So here is the answer of Douglas E. Comer, coauthor of the book, on the topic for the next fellow looking for the same answer:

It's an inside joke. The TCP standard says to choose a value other than 1.
My co-author, Dave Stevens, and I tossed a coin, and we used his birthday (September 4th). There are a couple of other fun items in the book (hint: look in the index).

Regards,
Doug Comer

After reading Mr. Comer reply, I have checked the index and there is a list of well-known constants such as the world renowned 1.3.6.1.2.1 !