Olivier Langlois's blog

It is an ok Gigabit router with its wireless functionality working fine. However when connecting a PS3 on the router and let it test the router capabilities, the router fails the UPnP testing. In my opinion, this is inexcusable when you sell a product at premium price and you market it as a "gaming" router, you would expect the company to have tested it with all the mainstream consoles.

I have contacted Cisco/Linksys customer support chat service to report the problem and get assistance and basically, I have been told something like that the problem was my fault and that I had to change some obscure and unrelated WiFi settings to fix my problem.

I was not satisfied by the answer and I decided to see for myself why the PS3 is reporting a UPnP failure with that router.

2 tools have been required for the analysis.

1. Wireshark
2. An Ethernet hub

A hub is a piece of networking hardware allowing you to share an Ethernet link between more than 2 network devices. Those were common in the 90s when processing power and memory were expensive. Today they have been replaced by the more robust and powerful Ethernet switches. My setup has been to plug my PC between the Linksys router and my PS3 and let Wireshark running on my PC sniff the UPnP exchange between them.

Before going in the details of the PS3 UPnP test, I want to share some interesting details about the whole PS3 Internet testing which is close to be undocumented on the Internet

Testing Internet Connection:

It does it by doing a HTTP request on fus01.ps3.update.playstation.net.


GET /update/ps3/list/us/ps3-updatelist.txt HTTP/1.1
Host: fus01.ps3.update.playstation.net
Connection: Keep-Alive
Accept-Encoding: identity
User-Agent: PS3Update-agent/1.0.0 libhttp/1.0.0

HTTP/1.1 200 OK
Server: Apache
ETag: "4d8c4dbf774c6349ad778577e53bd8c7:1285037260"
Last-Modified: Tue, 21 Sep 2010 02:47:40 GMT
Accept-Ranges: bytes
Content-Length: 252
Content-Type: text/plain
Date: Mon, 06 Dec 2010 03:22:33 GMT
Connection: keep-alive

# US
Dest=84;CompatibleSystemSoftwareVersion=3.5000-;
Dest=84;ImageVersion=0000b437;SystemSoftwareVersion=3.5000;CDN=http://dus01.ps3.update.playstation.net/update/ps3/image/us/2010_0921_0215e26d1dadeb950471a9c3397a140a/PS3UPDAT.PUP;CDN_Timeout=30;


Test PSN connection:

It does so by establishing an HTTPS connection with auth.np.ac.playstation.net. Obviously since it is encrypted, I cannot comment much about it.

NAT Type testing.

The PS3 sends a series of STUN (Session Traversal Utilities for NAT) (I am mentionning STUN in another blog entry for the curious) requests to us.np.stun.playstation.net

So now, lets get back to the UPnP testing. Here is a short overview of the whole procedure:

• Broadcast a request to find all network devices that are a InternetGatewayDevice
• Query its capability by requesting a XML file
• If the service WANIPConnection is supported, continue the test
• Call GetExternalIPAddress to obtain the router public Internet IP address
• Call AddPortMapping to open a public port on which all inbound traffic will be forwarded to the PS3

Here is how goes the exchange with the WRT330N router:


PS3 Broadcast:

M-SEARCH * HTTP/1.1
HOST: 239.255.255.250:1900
MAN: "ssdp:discover"
MX: 5
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1

2 seconds timeout. Retransmit the same request

After about 1.5 seconds after the 2nd retransmission, the router finally reply:

HTTP/1.1 200 OK
EXT:
SERVER: ipOS/7.2, UPnP/1.0, ipSSDPDevice/1.0
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
LOCATION: http://192.168.1.1/root.sxml
CACHE-CONTROL: max-age=1800
USN: uuid:1C7AE0B4-AF9D-3FD9-AC27-04FAE3357DD5::urn:schemas-upnp-org:device:InternetGatewayDevice:1
Content-Length: 0

PS3 request this:

GET /root.sxml HTTP/1.1
HOST: 192.168.1.1:80

PS3 UPnP request:

POST /wipconn HTTP/1.1
HOST: 192.168.1.1:4444
Content-Length: 290
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#GetExternalIPAddress"

Linksys Router:

HTTP/1.1 200 OK
SERVER: ipOS/6.8 UPnP/1.0 IGD/1.0
EXT:
Transfer-Encoding: Chunked

197
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><soap:Body><u:GetExternalIPAddressResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"><NewExternalIPAddress>xxx.xxx.xxx.xxx</NewExternalIPAddress></u:GetExternalIPAddressResponse></soap:Body></soap:Envelope>
0

2nd PS3 UPnP request:

POST /wipconn HTTP/1.1
HOST: 192.168.1.1:4444
Content-Length: 644
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping"

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:AddPortMapping xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
<NewRemoteHost></NewRemoteHost>
<NewExternalPort>3658</NewExternalPort>
<NewProtocol>UDP</NewProtocol>
<NewInternalPort>3658</NewInternalPort>
<NewInternalClient>192.168.1.109</NewInternalClient>
<NewEnabled>1</NewEnabled>
<NewPortMappingDescription>192.168.1.109:3658 to 3658 (UDP)</NewPortMappingDescription>
<NewLeaseDuration>0</NewLeaseDuration>
</u:AddPortMapping>
</s:Body>
</s:Envelope>

Linksys Router:

HTTP/1.1 200 OK
SERVER: ipOS/6.8 UPnP/1.0 IGD/1.0
EXT:
Transfer-Encoding: Chunked
151
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><soap:Body><u:AddPortMappingResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"></u:AddPortMappingResponse></soap:Body></soap:Envelope>
0


By looking the exchange it looks all good and in the router log, there is even an entry that shows that the Port mapping occured:


[INFO] Sun Dec 05 20:40:09 2010 UPnP renew entry 255.255.255.255 <-> 24.37.208.168:3658 <-> 192.168.1.109:3658 UDP timeout:-1 '192.168.1.109:3658 to 3658 (UDP)'


I initially believed that the error is the returned length of the router reply for the AddPortMapping request. It reports a length of 151 bytes but if you count them, I come to something around 340 chars! The GetExternalIPAddress call has the same problem. The reported size is 197 while in fact, I count around 410 chars!

However, a coworker of mine pointed out to me that the chunk size values are in hex as described in the RFC 2616. So I can only conclude that the problem comes from the PS3 that does not handle correctly HTTP chunked transfer coding in the context of UPnP exchange.

Another indication that the chunked transfer encoding in the replies is the culprit of the problem, it is that at the end of both exchange between the PS3 and the WRT330N is that the PS3 is sending back TCP RST segments to the router which means that the PS3 is closing its connection with the router before the router having finished to send its replies.

For your reference, here is an exchange with my new NetGear router, the WNDR37AV that works like a charm!


In less than 2 ms the reply to the M-SEARCH query comes back.

HTTP/1.1 200 OK
Cache-Control: max-age=1800
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
USN: uuid:12345678-0000-0000-0000-00000000abcd::urn:schemas-upnp-org:device:InternetGatewayDevice:1
EXT:
Server: Linux/2.6.15-1.2054_FC5 UPnP/1.0 miniupnpd/1.0
Location: http://192.168.1.1:5555/rootDesc.xml

PS3 request this:

GET /rootDesc.xml HTTP/1.1
HOST: 192.168.1.1:5555

PS3 UPnP request:

POST /ctl/IPConn HTTP/1.1
HOST: 192.168.1.1:5555
Content-Length: 290
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#GetExternalIPAddress"

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:GetExternalIPAddress xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
</u:GetExternalIPAddress>
</s:Body>
</s:Envelope>

Netgear reply:

HTTP/1.1 200 OK
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 356
Server: Linux/2.6.15-1.2054_FC5 UPnP/1.0 miniupnpd/1.0
Ext:

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><s:Body><u:GetExternalIPAddressResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"><NewExternalIPAddress>xxx.xxx.xxx.xxx</NewExternalIPAddress></u:GetExternalIPAddressResponse></s:Body></s:Envelope>

2nd PS3 UPnP request:
POST /ctl/IPConn HTTP/1.1
HOST: 192.168.1.1:5555
Content-Length: 644
Content-Type: text/xml; charset="utf-8"
SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping"

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:AddPortMapping xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
<NewRemoteHost></NewRemoteHost>
<NewExternalPort>3658</NewExternalPort>
<NewProtocol>UDP</NewProtocol>
<NewInternalPort>3658</NewInternalPort>
<NewInternalClient>192.168.1.109</NewInternalClient>
<NewEnabled>1</NewEnabled>
<NewPortMappingDescription>192.168.1.109:3658 to 3658 (UDP)</NewPortMappingDescription>
<NewLeaseDuration>0</NewLeaseDuration>
</u:AddPortMapping>
</s:Body>
</s:Envelope>

Netgear reply:

HTTP/1.1 200 OK
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 260
Server: Linux/2.6.15-1.2054_FC5 UPnP/1.0 miniupnpd/1.0
Ext:

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><s:Body><u:AddPortMappingResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"/>
</s:Body>
</s:Envelope>


In conclusion, I am disapointed by the quality of the Linksys product and the lack of attention of details that the company did put in the QA before the release of that product since the UPnP bug could have been caught by a very simple test in less than 5 minutes. Apparently the bug is in the PS3 that does not support UPnP interaction with HTTP replies using chunked transfer coding but I was expecting to purchase a router compatible with my PS3 when I bought a gaming router. For that reason, I prefer the NetGear router.

Happy networking!

I have posted on my website a small C++ program that I have been asked to write during an interview with Facebook at Fall 2009. One of their interview was related to graph theory and the problem was to find the the shortest distance between to nodes in a graph. The best algorithm to use to solve this problem is the breadth-first traversal algorithm.

You can look at the source code here at:
http://www.olivierlanglois.net/archive/graph_cpp.htm

and you can read more about the breadth-first traversal algorithm in Robert Sedgewick excellent book on algorithms.

Today was my last day at StreamTheWorld. I spent the last 3 years there as the company C++ Tech Lead in charge of the company C++ streaming servers development. It has been a nice ride filled with challenges.These servers are used by over 1600 radio stations including AOL/CBS radios accessible through Winamp and Flash players.

Among my various realizations. There are:

• Scale and improve the performance and the reliability of the company audio/video streaming servers to the point where it can concurrently handles hundreds of thousands of live feeds every day in more than 25 countries, including during peak audience dayparts. The reliability and the quality of the server software have been proven to such industry leaders as CBS, Entercom, Multimedios, AOL and more. This represents 20x improvement over the original capacity.
• Added more functionality to the server by growing the server code base from 25K lines in June 2007 to 200K lines in only 2 years while improving robustness and performance.
• Designed and implemented a high performance targetted stream ad insertion system that serve end-user ads based geoip lookup. This system is now used by ESPN Radio on Internet.
• Design and implement a protocol stack to support Adobe proprietary protocol RTMP clients
• Add AAC support to the streaming server
• Improve Microsoft MediaPlayer and Silverlight support.

I will start my new job Monday next week. Visit back my blog or my LinkedIn profile if you are curious to learn for which amazing new employer I will be working for.

It is a very interesting book. Anyone having been involved in a large scale software development projects will recognize typical problems often seen in this type of projects while reading this book and by experience we have developed an intuition about how to deal with the complexity inherent to large scale projects. Mr Lakos book is the first, to my knowledge, that address this subject and it formalize extremely well the problems of large scale projects and propose a methodology and principles to apply to keep the development and maintainability of these projects manageable. It is a very refreshing C++ book and I recommend it to any C++ professionals not having read it yet.

The most interesting chapters in my opinion are chapter 5 and chapter 6. Chapter 5 presents refactoring methods to remove cyclical dependencies among components, reduce inter dependencies and how to layout components into software layers that will ease reusability and maintainability. Chapter 6 discuss about component insulation. Its difference with encapsulation, the benefits of component insulation and its cost and when it is wise to avoid insulation.

While I was integrating a fix to a bug that has been reported to me by Daniel Quadros in my primality testing C++ module and also adding gcc asm support to it. I used that opportunity to rework my C++ source code to HTML converter. The 2 things that I added to the converter are:

• replace the usage of the <font> tags with <span> tags in combination of CSS classes
• size optimization of the converted document

The size optimization came from the very simple observation that my converter was stateless and was not recognizing when multiple consecutive sections sharing the same colors were processed. I just added a small FSM to my converter to keep track of the current state and the converter now close the current tag only if there is a transition to a new color. The most impressive result came from my DCEL C++ file with a 25% file size reduction!

Also, I am just starting to appreciate to power of CSS. I will now be able to change the code snippet colors across my whole website by just updating a small CSS file. As a side bonus by having switched from the <font> tags to <span>, I can now use my C++ highlight feature in the code snippets contained in this blog as the blog engine was forbiding using <font> tags on posts but allows <span>.

I just started to read this book which is an updated version of Practical Cryptography. This book is coauthored by Bruce Schneier. From what I have seen so far. It looks like a very nice introduction to cryptography that is very accessible. For more in depth coverage on cryptography, I would recommend looking at Applied cryptography. I will probably have more to say about the book when I am done reading it.

This book wants to be the encyclopedia of networking science. It is very complete and covers a lot of topics. In my opinion, this is not very useful because you either read a chapter covering a topic that you already totally master and hence learn nothing or the topic is totally new and the book barely scratches the surface of the topic letting many questions unanswered.

It is a very complete guide. It did allow me to understand my own blog CSS code and I have been able to fix an alignment problem related to margin values in <div> tag style property. Since my banner was displaying correctly in IE (incorrect behavior) but was misaligned in Firefox, a full understanding of CSS was necessary to figure out what was wrong and this book helped a lot. This small experience did allow me to appreciate the type of challenges HTML professionals are confronted to daily to support a wide range of browsers.

My only complain, and maybe it is because HTML editing is not my thing, is that while I was reading the book, I did not felt the author has succeeded in transmitting his passion for the topic. Instead reading this book felt as exciting as reading a dictionary for the most part. For this reason, I would not recommend it for learning CSS but if you need a good reference document then it might do fine.

I have submitted a patch for the bugzilla bug 3606:

http://bugzilla.dre.vanderbilt.edu/show_bug.cgi?id=3606

Fix function ACE::handle_ready + SOCK_Test modif to test the fix

The patch include the following:

1. Fix for the bug 3606 in function ACE::handle_ready
2. Modif of the unittest SOCK_Test to validate the fix
3. Removal of the define ACE_HAS_LIMITED_SELECT

There has been a very long thread about it last summer on the ACE mailing list and I was arguing that it was dangerous to use select() when poll() is available because if you pass an handle whose value that is higher than FD_SETSIZE (typically 1024), it causes a buffer overflow on the stack which is hard to debug.

At the end, a consensus emerged that ACE_HAS_LIMITED_SELECT should be removed. Actually if some platform would benefit from that, it would be safer to define ACE_HAS_UNLIMITED_SELECT instead.

4. Remove the duplicated code from ACE::handle_ready() at different locations to replace it with the appropriate ACE namespace function call.

For those who do not know what RAII is, you are probably using it without knowing it. One common use of this C++ idiom is to keep code manipulating mutexes exception safe. While I was writing a thread safe getter with a scoped lock like this:

A B::getA() const
{
  ScopedLock sc(m_lock);
  return m_a;
}

A doubt sparkled in my mind about whether or not the object copy performed by the return statement was protected by the lock. It must be since I have seen tons of functions using similar pattern but I could not explain why. I asked to another senior developer and he could not tell neither. By the way, this is what I love about the C++ language. You can have years of experience with it, there are millions of small details and any day, you may stumble on a mind boggling detail that force you to stop and think about it.

So if we come back to the original question, I guess that I could have checked what the standard document says about it but it is boring to do so. I prefer to do small scientific experiments to figure out myself what is the answer. So I wrote this small program:

#include <iostream>

class RAIIObj
{
  public:
  RAIIObj() { std::cout << "RAIIObj()\n"; }
  ~RAIIObj() { std::cout << "~RAIIObj()\n"; }
};

class A
{
  public:
  A() {}
  A( const A & ) { std::cout << "A( const A & )\n"; }
};

class B
{
  public:
  B() {}
  A getSyncAByVal() const;
  private:
  A m_a;
};

A B::getSyncAByVal() const
{
  RAIIObj scopedLock;
  return m_a;
}

int main( int argc, char *argv[] )
{
  B b;
  A retVal = b.getSyncAByVal();
  return 0;
}

and here is the output confirming that the getter code is correct:

RAIIObj()
A( const A & )
~RAIIObj()

Now that I know the answer, I have found another proof confirming that return statement object copies are performed before the local lock goes out of scope. If you were returning by value a local object, it absolutely must be copied before going out of scope.

Upon reception of RST segment, the receiving side will immediately abort the connection. This statement has more implications than just meaning that you will not be able to receive or send any more data to/from this connection. It also implies that any unread data still in the TCP reception buffer will be lost. This information can be found in TCP/IP Internetworking volume 2 and Unix Network Programming Volume 1 third edition but in my opinion those books do not put enough emphasis on that detail and if you are not reading these books to find this exact detail, it might slip away from your attention.

Now, what are the conditions to receive a RST segment? The easiest way is to enable the SO_LINGER option with a timeout value of 0 on a server socket. As soon as the connection will be closed by the server, it will send a RST. You can learn more about SO_LINGER in the book Unix Network Programming Volume 1 third edition. The other possibility is if a server receives data after having closed the connection. If you enable the LINGER option, sending RST is the expected behavior but you can easily be bitten by the second possibility and here is an example on how it can happen.

Imagine a HTTP server that limits the number of simultaneous connections. Upon accepting a new connection, it might compare a global connection counter against a configured limit and if the counter has reached the limit then immediately send back a 503 error and close the connection. Do you see the problem? The server is closing the connection before receiving the HTTP request. There is a high probability that the client will never receive the 503 reply because a RST segment will immediately follow the reply.

A not too good solution would be to call shutdown(SHUT_WR) before closing the socket. What this will do is force the server to send a FIN segment before the RST segment. I have tried this solution and this seems to greatly improve the probability that the client application will receive the data before receiving the RST segment.

A better way is to program the server to read the request even if it has no intention to actually process it. This ensures that no RST will be sent unintentionally.

There remains one question. What exactly means data reception on a closed connection? Does that only include new received TCP segment or could data still in the TCP receive buffer of a closing connection be considered as data arrived after the connection has been closed? My intuition is that this question is open to interpretation and the TCP behavior will vary from one platform to the other. However, even if your platform does not react with a RST if a TCP connection receive buffer is not empty when it is closing it; my opinion is that you should not rely on that since this is a race condition. You could be closing the connection before or after receiving the client request depending on the RTT (Round Time Trip) of the connection.