As a conclusion to the previous 3 posts on the topic of TCP tuning, this post describes how to adjust the TIME_WAIT value in tcp.h
The issues all point to one problem: the number of ports in use. On the default linux kernel, an established connection gets, after it has been used, in some idle state. This is to ensure that no lost packages will reach a newly established connection to a different client. The time after which such a connection gets dropped should be 4 minutes, according the the TCP reference files. In our linux kernels, it is set to 60 seconds. This concept basically dates from the beginning of TCP, when computers sucked big time.
To improve the number of possible concurrent connections, I have specified a larger port range:
net.ipv4.ip_local_port_range = 1024 65535
This used to be something like 32768 to 61000. So this gives me a factor 2 improvement. I came to the conclusion that a factor 2 might not be enough, looking to the future. Our site is still growing, so another way to improve performance of the servers had to be found.
The 65k possible concurrent connection limit I now have actually is the maximum number, due to the fact that TCP only allows 16 bit (65536) ports. So the lifetime of a connection as such should be reduced. This is a tricky concept. The TIME_WAIT number is hardcoded into the linux kernel. A kernel rebuild is required.
Using this tutorial, I could remotely compile a new linux kernel, and named it “2.6.24.3-cautela”. To decrease the TIME_WAIT number, I edited:
/usr/src/linux/include/net/tcp.h
as explained in this document.