[rabbitmq-discuss] Getting maximum performance from RabbitMQ cluster

Tue May 6 05:16:23 BST 2014

Today I've tested the overhead that "x-consistent-hash" exchange adds. In
this test setup I had 16 queues with same weights behind hash exchange (by
custom hash-header). All queues declared on same node, non-HA, publisher
threads connecting to that node as well. Throughput numbers

1 thread - 9K/s
2 threads - 18K/s
4 threads - 40K/s
8 threads - 82K/s
16 threads - 95K/s

Publish rate is reduced by about a half, Rabbit CPU at 16 threads was
constantly 99%+ busy.

The more I think about scaling RabbitMQ, the more I come with conclusion
that cluster is no help. In the test case above for example it would not
help at all to have a cluster as publishers have no means to know which node
the message would end up going to and thus can't pick which node to connect
for most efficient publishing (to avoid cross node synchronization
overhead). It would be better to have multiple independent RabbitMQ brokers,
each having multiple consumers with even load distribution (via hash
exchange) and publishers randomly choosing which node to connect to.

Am I going crazy here or  clustering feature
<https://www.rabbitmq.com/clustering.html>   of RabbitMQ is suited best for
reliability and not scalability? I just don't see how it helps so far.

--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Getting-maximum-performance-from-RabbitMQ-cluster-tp35347p35358.html
Sent from the RabbitMQ mailing list archive at Nabble.com.