[rabbitmq-discuss] Huge latency in Linux, compared with Leopard

[Holger Hoffstätte]

Matthias Radestock wrote:
> Adding the line
> 
>     -kernel inet_default_listen_options '[{nodelay, true}]' \
> 
> to the options in the rabbitmq-server script does improve the results
> significantly.

Yay! Glad it helped.

> So should we make TCP_NODELAY the default?

That's difficult to decide, though IMHO it is probably the right thing to
do for most common scenarios today. Ironically after I had found the
problem I remembered that I had read something related on the Linux kernel
list, where someone from RedHat was complaining about 40ms latency. :)

Find the juicy details here:
http://marc.info/?l=linux-netdev&m=122091100407930&w=2

Other treads of interest may be:
http://www.nabble.com/slow-tcp-acks-on-loopback-device-td464386.html
http://www.nabble.com/What-happened-to-CONFIG_TCP_NAGLE_OFF--td7588964.html

I also found that the Windows stack automatically turns Nagle off for
loopback connections, and I assume other BSD-based OS (OS X) do as well.
This would explain the platform-specific observations.

Nagle makes sense in order to maximize bandwidth and prevent wasting
packets for small payloads (~couple bytes, as for telnet) that can be
packed together into one MTU. This may or may not make sense for Rabbit
but is dependent on the network link and the payloads. So turning it off
may be the right thing for localhost/LAN messaging where the cost of a
single packet is low, but not at all for large(r) payloads and high
bandwidth or slower tranports. I doubt that there are any people left who
want to run over modem or carrier pidgeon..

I am really looking forward to SCTP (as the lkml thread mentions) since so
many of the underlying assumptions behind TCP have changed these days that
no matter what the stacks try to do, it's always kind of wrong. Now it
only needs to become stable..

Holger