[rabbitmq-discuss] Scaling Throughput with Highly Available Queues

[Richard Raseley]

Matthias,

Imagine the following scenario:

There are four servers named rmq01, rmq02, rmq03, and rmq04. Servers rmq01
and rmq02 are clustered; servers rmq03 and rmq04 are clustered (so we now
have 2x two node clusters).

On both clusters we create a vhost named "default", a direct exchange named
"default", a queue named "default", a user "default" (which has read, write
and config access to the aforementioned objects) with a password of
"default" and apply the ha-all mode to the "default" vhost.

We place these two identically configured clusters inside of a load
balanced pool with a "least queue depth" configuration. We then configure
the consumers and producers associated with this app to connect to the VIP
(which maps to the load balanced pool) to send messages to the "default"
exchange and consume messages from the "default" queue.

Assuming we have more producers and consumers than we do RabbitMQ nodes
(and they get distributed properly), we've now increased our throughput -
as we now have two master queue processes to work with on this "single"
queue.

Regards,

Richard


On Tue, Mar 19, 2013 at 2:59 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> On 19/03/13 21:55, Richard Raseley wrote:
>
>> The relevance lies in the fact that the design of this particular
>> application component relies on a single highly available queue bound to
>> a single exchange. My thought (which you've mostly confirmed) was that
>> if I create multiple clusters with mirrored configurations (e.g. same
>> exchanges, queues, users, etc.) and place those in a single load
>> balanced pool I will be able to scale my performance (while maintaining
>> high availability for all messages) whereas if I add additional nodes to
>> a cluster I will not - as all operations will happen via the mater queue
>> process which resides on exactly one node.
>>
>
> But your "*single* highly available queue", can only live in one cluster,
> since it's a *single* queue. So I don't see how extra clusters help here.
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130319/af1f5ce8/attachment.htm>