[rabbitmq-discuss] Poor performance using a single RabbitMQ connection on high-latency networks

[Matthew Sackman]

On Sat, Feb 06, 2010 at 04:32:35AM +0100, Holger Hoffstaette wrote:
> This rabbit hole goes deep..

Indeed it does. I repeated all my tests this morning with the buffers
all hard set to 65536 and things got slightly better with large
messages (~1kB), but worse with small messages.

I then set up a high latency loopback:
# tc qdisc add dev lo root netem delay 100ms 20ms distribution pareto loss 0.01%

So a 100ms delay, ±20ms on a pareto distribution, with 0.01% packet
loss.

With our default buffer sizes:
rabbitmq-java-client/build/dist$ sh runjava.sh com/rabbitmq/examples/MulticastMain -a -z 60 -i 10 -s 1024 -r 12000
starting consumer #0
starting producer #0
recving rate: 8 msg/s, min/avg/max latency: 207432/3279948/6437079 microseconds
sending rate: 322 msg/s
recving rate: 13 msg/s, min/avg/max latency: 6437120/11372211/16522991 microseconds
sending rate: 14 msg/s
recving rate: 14 msg/s, min/avg/max latency: 16522969/21456014/26551356 microseconds
sending rate: 13 msg/s
recving rate: 13 msg/s, min/avg/max latency: 26551374/31650558/36685934 microseconds
sending rate: 13 msg/s
recving rate: 13 msg/s, min/avg/max latency: 36687449/41818269/46842999 microseconds
sending rate: 13 msg/s
recving rate: 13 msg/s, min/avg/max latency: 46851173/51931717/56934090 microseconds
recving rate avg: 13 msg/s
sending rate avg: 67 msg/s

With buffers set to 65536 and 65536:
recving rate: 180 msg/s, min/avg/max latency: 220489/3300700/6318430 microseconds
sending rate: 648 msg/s
recving rate: 286 msg/s, min/avg/max latency: 6318355/10433052/12858930 microseconds
sending rate: 256 msg/s
recving rate: 264 msg/s, min/avg/max latency: 12676703/13635080/14137168 microseconds
sending rate: 294 msg/s
recving rate: 282 msg/s, min/avg/max latency: 12437468/13308885/14238964 microseconds
sending rate: 275 msg/s
recving rate: 272 msg/s, min/avg/max latency: 12971076/13367869/13966905 microseconds
sending rate: 262 msg/s
recving rate: 251 msg/s, min/avg/max latency: 13644724/14051244/14746586 microseconds
recving rate avg: 265 msg/s
sending rate avg: 332 msg/s

With no buffers set (thus allowing auto resizing to take place):
recving rate: 1012 msg/s, min/avg/max latency: 196813/2498703/3712777 microseconds
sending rate: 6146 msg/s
recving rate: 6104 msg/s, min/avg/max latency: 2782802/4983136/5602666 microseconds
sending rate: 10466 msg/s
recving rate: 9649 msg/s, min/avg/max latency: 5440633/5971475/6431049 microseconds
sending rate: 10637 msg/s
recving rate: 9970 msg/s, min/avg/max latency: 6204889/6718248/7106264 microseconds
sending rate: 6681 msg/s
recving rate: 9535 msg/s, min/avg/max latency: 3934919/4661861/7229773 microseconds
sending rate: 10795 msg/s
recving rate: 8960 msg/s, min/avg/max latency: 4055794/4703495/5892391 microseconds
recving rate avg: 7755 msg/s
sending rate avg: 9121 msg/s

Needless to say, my flabber has been ghasted. Many thanks for bringing
this issue to our attention. Unless there are further revelations, it's
very likely we'll remove this setting in the next release.

Matthew