[rabbitmq-discuss] Poor performance using a single RabbitMQ connection on high-latency networks

[Holger Hoffstaette]

On Tue, 02 Feb 2010 12:35:12 +0000, Matthias Radestock wrote:

> Matthew Sackman wrote:
>> I wonder whether Matthias or Tony may recall why our server options are
>> set as they are.
> 
> We don't want to set very large buffer sizes since we do not want to
> increase the memory footprint of connections unnecessarily. The figures we
> currently use are the minimum numbers above which I saw no significant
> performance improvements in the tests I ran ~2 years ago. But those tests
> weren't very extensive.

Fair enough, but I'm not sure why the values have to be even smaller than
e.g. Linux' vanilla defaults. Ideally - and given that Rabbit is mostly
used on Linux/Unix - they should not be necessary at all, especially as
they prevent any client-side settings from taking effect (I think - would
have to check with wireshark).

But in principle you're right, there are so many first- and second-order
effects and OS/network permutations at play here that it's impossible to
find the one perfect default. XP is different from Vista from Win7, and
RedHat's kernels have little in common with the mainline release. Even the
latest few mainline releases have seen an insane number of back-and-forth
fixes for all sorts of TCP behaviour, timers, CPU scheduling and whatnot.

That's why I think the defaults should ideally just be removed.
Platform/language-specific tuning recommendations can then be created for
different usage scenarios on the wiki by any users on their respective
platforms..provided it's easy enough to arrive at meaningful numbers and
submit them.

The comment re. memory usage with a lot of connections is interesting
though - is this really a problem even with epoll (aka kernel polling)?
What would be a large number of connections - 100? 1000?

I cannot really afford to take a few weeks off just for this, even if it
is an interesting project. Can't you get an intern or student ;)

Holger