[rabbitmq-discuss] Mirrored queue performance factors
paul.bowsher at gmail.com
Tue May 21 20:12:41 BST 2013
We operate two dual-node brokers, each broker having quite different queues
and workloads. Each box has 24 cores (H/T) worth of Xeon E5645 @ 2.4GHz
with 48GB RAM, connected by Gigabit LAN with ~150μs latency, running RHEL
5.6, RabbitMQ 3.1, Erlang R16B with HiPE off. We've tried with HiPE on but
it made no noticeable performance impact, and was very crashy.
We appear to have hit a ceiling for our message rates of between 1,000/s
and 1,400/s both in and out. This is broker-wide, not per-queue. Adding
more consumers doesn't improve throughput overall, just gives that
particular queue a bigger slice of this apparent "pool" of resource.
Every queue is mirrored across the two nodes that make up the broker. Our
publishers and consumers connect equally to both nodes in a persistant way.
We notice an ADSL-like asymmetry in the rates too; if we manage to publish
a high rate of messages the deliver rate drops to high double digits.
Testing with an un-mirrored queue has much higher throughput, as expected.
Queues and Exchanges are durable, messages are not persistent.
We'd like to know what we can do to improve the situation. The CPU on the
box is fine, beam takes a core and a half for 1 process, then another 80%
each of two cores for another couple of processes. The rest of the box is
essentially idle. We are using ~20GB of RAM in userland with system cache
filling the rest. IO rates are fine. Network is fine.
Is there any Erlang/OTP tuning we can do? delegate_count is the default 16,
could someone explain what this does in a bit more detail please?
Any thoughts anyone has on this would be much appreciated.
Senior Software Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss