[rabbitmq-discuss] Mirrored queue performance factors

[Paul Bowsher]

Hi,

We operate two dual-node brokers, each broker having quite different queues 
and workloads. Each box has 24 cores (H/T) worth of Xeon E5645 @ 2.4GHz 
with 48GB RAM, connected by Gigabit LAN with ~150μs latency, running RHEL 
5.6, RabbitMQ 3.1, Erlang R16B with HiPE off. We've tried with HiPE on but 
it made no noticeable performance impact, and was very crashy.

We appear to have hit a ceiling for our message rates of between 1,000/s 
and 1,400/s both in and out. This is broker-wide, not per-queue. Adding 
more consumers doesn't improve throughput overall, just gives that 
particular queue a bigger slice of this apparent "pool" of resource.

Every queue is mirrored across the two nodes that make up the broker. Our 
publishers and consumers connect equally to both nodes in a persistant way. 
We notice an ADSL-like asymmetry in the rates too; if we manage to publish 
a high rate of messages the deliver rate drops to high double digits. 
Testing with an un-mirrored queue has much higher throughput, as expected. 
Queues and Exchanges are durable, messages are not persistent.

We'd like to know what we can do to improve the situation. The CPU on the 
box is fine, beam takes a core and a half for 1 process, then another 80% 
each of two cores for another couple of processes. The rest of the box is 
essentially idle. We are using ~20GB of RAM in userland with system cache 
filling the rest. IO rates are fine. Network is fine.

Is there any Erlang/OTP tuning we can do? delegate_count is the default 16, 
could someone explain what this does in a bit more detail please?

Any thoughts anyone has on this would be much appreciated.

Many thanks,

Paul Bowsher
Senior Software Engineer
Globaldev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130521/92795d18/attachment.htm>