[rabbitmq-discuss] Mirrored queue performance factors

[Simon MacMullen]

On 21/05/13 20:12, Paul Bowsher wrote:
> Hi,

Hi.

> We operate two dual-node brokers, each broker having quite different
> queues and workloads. Each box has 24 cores (H/T) worth of Xeon E5645 @
> 2.4GHz with 48GB RAM, connected by Gigabit LAN with ~150μs latency,
> running RHEL 5.6, RabbitMQ 3.1, Erlang R16B with HiPE off. We've tried
> with HiPE on but it made no noticeable performance impact, and was very
> crashy.
>
> We appear to have hit a ceiling for our message rates of between 1,000/s
> and 1,400/s both in and out. This is broker-wide, not per-queue. Adding
> more consumers doesn't improve throughput overall, just gives that
> particular queue a bigger slice of this apparent "pool" of resource.
>
> Every queue is mirrored across the two nodes that make up the broker.
> Our publishers and consumers connect equally to both nodes in a
> persistant way. We notice an ADSL-like asymmetry in the rates too; if we
> manage to publish a high rate of messages the deliver rate drops to high
> double digits. Testing with an un-mirrored queue has much higher
> throughput, as expected. Queues and Exchanges are durable, messages are
> not persistent.

Just to check - how much faster is your unmirrored throughput?

But those performance figures sound wrong, I can go much faster than 
that on a less powerful machine.

How large are messages? Are you using acks / confirms?

Our Java client ships with a performance measuring tool. It would be 
good to test with that so that we are running the same tests. Can you 
grab the Java client tarball, unpack it and run:

$ ./runjava.sh com.rabbitmq.examples.MulticastMain -a -h 
amqp://<server-name>

and tell us what message rates you get? That command line should use 
tiny unpersistent / unacked / unconfimed messages, so it should give us 
a baseline to work with.

If not all queues are configured to be mirrored you might need to add 
"-u <name of a queue to create that will be mirrored>".

> Is there any Erlang/OTP tuning we can do? delegate_count is the default
> 16, could someone explain what this does in a bit more detail please?

delegate_count is not itself used by mirroring; it (approximately) 
provides an upper bound on the parallelism available for cross-cluster 
channel -> queue message routing. It's pretty rare to want to change it.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal