[rabbitmq-discuss] questions about RabbitMQ linear scalability

[Simon MacMullen]

On 08/08/2013 11:49AM, Junius Wang wrote:
> Our team are performing some performance tests and we try to prove that
> we can get linear scalability of our services, it means when we add more
> RabbitMQ nodes, the throughputs should increase as well.  We use
> RabbitMQ as our backend message system. However, when we add more
> RabbitMQ servers to cluster, the throughput decreases which is not
> expected and I would like to ask for your help. Following are the
> details, any comments are appreciated.

<snip>

> We can see that, while two rabbitmq servers are used, we see no
> throughput improvement. This doesn’t make sense to me. To my
> understanding,  Here comes my questions:

The issue is that an individual unmirrored queue cannot scale out across 
more than one node in a cluster (and it's hard to see how it could do 
while maintaining ordering guarantees etc).

> 1.What does RabbitMQ cluster scalability means exactly? Does it refer to
> the capacity of handing messages? The queue capacity? Or the message number?

Primarily more cluster nodes can either give more redundancy, or 
capacity for more queues. They cannot make individual queues have more 
capacity.

> 2.The  mirrored queues seems to affect performance in a large scale.
> What’s the possible cause of this?

When you have an unmirrored queue it needs to do a certain amount of 
work on a certain node.

When you make that queue mirrored, it needs to do the same amount of 
work on that node, and also the same amount of work on the other nodes. 
But in addition to that it also needs to do more work on each node, to 
keep the mirrors informed of what each other are doing. This is not a 
trivial amount of work!

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal