[rabbitmq-discuss] Architecture Question

Tue May 28 05:49:16 BST 2013

Hello all!

I've been running some tests locally, trying to determine whether RabbitMQ
will meet our performance requirements.

I'm running the broker (3.1.1) on a Dual Xeon E5 (12 cores total) with 256
MB and Windows Server 2012, on Erlang R16B (64bit). One of the primary
workflows that we want to enable is using a cluster  is to have several
workers behind the cluster taking messages from multiple clients.

With a single producer -> direct exchange -> single consumer on a single
broker, I can get about 38000 messages / second. (I'm not sure where the
performance limit is here, since I'm not queuing any messages, the consumer
is the equivalent of /dev/null, and the CPU on the broker doesn't go above
12%)

To test it, I ran two brokers on the same server, and tried setting up a
couple of different exchange/queue layouts:

   - Producers publishing to a single consistent hash exchange, with each
   consumer connecting to a private queue bound to the exchange
   - Producers publishing to a single exchange / default exchange, with
   each consumer connecting to a single queue off the exchange

In the first scenario above, if I use 24 consumers (and hence 24 queues)
distributed equally across both brokers, I can barely muster 52000 messages
/ second (and going up to 30% CPU on each Erlang broker)

In the second scenario, I stop at under 10000 messages / second.

This leads me to believe that the former is the better design, but what
kind of architecture would give a 4 fold or 10 fold increase in throughput?
Would we have to hash prior to RabbitMQ? An x-consistent-hash in front of
multiple x-consistent-hash exchanges?

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130527/e8fe4a4e/attachment.htm>