[rabbitmq-discuss] Doubt regarding rabbit servers in clustered environment

[Marek Majkowski]

On Sat, Nov 20, 2010 at 05:01, Kshitiz Garg <stephanion2002 at gmail.com> wrote:
> Since our 8 queues are going to handle lot of data, we can cluster 8
> RabbitMQ nodes, each to serve a single queue and its contents. But there are
> two concerns:

Remember that any modification of shared database must be agreed between
all the nodes in the cluster. 8 nodes is a lot and might result in a
latency for commands like exchange.declare or queue.bind. We rarely
see a cluster of more than a few nodes.

> 1. In the case a node (say, containing queue a) is down, how will the queue
> a's consumers will get their messages?

They won't.

> 2. A load balancer like HA Proxy in front of these nodes will not be of any
> use since nodes are not similar.

It would help to spread the connections equally between nodes.
That is pretty useful if you have a lot of connections.

> I am afraid that since we are neither getting "Load balancing" nor "High
> availability",  we will have to consider another broker.

For HA we're usually suggesting Active-Passive failover:
  http://www.rabbitmq.com/pacemaker.html
We're currently working on Active-Active HA, but it will take us some time.

Clustering does help in "Load balancing" but only in some situations.
For example it may be pretty useful for the 'fanout' cases, when you
have many queues.

But if you have a single queue, clustering won't help you.
It's a bit like with database sharding, if you have one monolitic table -
sharding won't help.

Cheers,
  Marek