[rabbitmq-discuss] Clustering

[Matthias Radestock]

Dmitriy,

Ben Hood wrote:
> Dmitriy,
> 
>> I set up a cluster of 2 nodes - rabbit at node1 and rabbit at node2.
>> Several producers will connect to either node and send a series of
>> messages with topic hierarchy as routing key (for example,
>> "a.b.c.d.e.f"). Consumers will also connect to either node and will
>> register to receive some subset of what is being sent - in other
>> words, for example, some of them will request "a.#" while others
>> might want to get only "#.e.f". My goal is to make it happen if at
>> least one of the nodes in the cluster is up and running. Both
>> producers and consumers will know about existence of both cluster
>> nodes, but they need to be able to randomly connect to either.

> As far as I know, message queues are not yet fully clustered in
> rabbitmq, meaning that you cannot yet transparently failover or
> migrate a queue to a different node in the cluster. These queues are
> visible from any node in the cluster however.

Ben is right about the (lack of) queue clustering. Whether or not that 
matters depends on what guarantees you expect on message delivery: Can 
you afford to lose messages? Is it ok for a message to be delivered to a 
consumer more than once?

You can guarantee exactly-once delivery if you use transactions, durable 
queues and exchanges, and persistent messages, but only as long as any 
failing node eventually recovers.

If you can afford to lose the odd message then the simplest setup is to 
just have consumers detect when connections goes down, connect to the 
other node and recreate their queues. Similarly, producers just 
re-connect to the second node on failure of the first node, and start 
sending messages to the same exchange as before.

If message loss is bad but duplication is ok then you can could set up 
two parallel routing topologies with producers sending messages to both 
nodes and each consumer consuming from two queues - one on each node.

There are various other options; which one is best really depends on the 
specific requirements of your application.


All this is going to improve significantly in AMQP 0-10, where there is 
a notion of a "session" whose lifecycle is independent from that of the 
transport connection. Sessions can transparently reconnect to other 
nodes and have built-in support for exactly-once delivery. We'd still 
need to add queue clustering to RabbitMQ, but that's on our roadmap 
already anyway.


Regards,

Matthias.