[rabbitmq-discuss] questions about RabbitMQ linear scalability

[Michael Klishin]

Junius Wang:

> 1.       The throughput of two node cluster is 50%-60% worse than a single node broker.

With mirroring, messages have to be copied to multiple (N or, depending on configuration, even all) nodes in the cluster. That obviously takes more time than not copying anything.

> 2.       Adding more node did have improvement on throughput but we only got 25% improvement(throughput of 3 node cluster is 25% better than 2 node cluster. 4 node cluster is 25% better than 3 node cluster too). What we expected is a 45-degree line, that means when 2 nodes are used the throughput is double. With 3 nodes, then triple.

You are not providing any details about your workload but the way you extend capacity with RabbitMQ is
by adding nodes and using more queues. Queue mirroring is an HA feature, which means copying
more data across the cluster.

Maximum throughput and highest availability are largely at odds with each other, so
you need to

1. Use multiple queues (and the number should grow with the number of nodes)
2. Choose what queues to mirror. Likely not all queues are equally important to your system, so you can make some of them non-HA.
3. Configure mirroring to, say, 2 nodes instead of "all".
4. If you know what node is the master for a particular queue (e.g. was declared on that node),
     make your clients connect there. It will decrease intra-cluster traffic.
--
MK

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130822/089f4ca1/attachment.pgp>