[rabbitmq-discuss] Clustering Behaviour

[Tim Watson]

Hi Max,

On 18 Aug 2013, at 13:34, Max Manders wrote:
> If I configure the cluster to use mirrored queues, I'm also working on the assumption that nodes may be transient and might get terminated and created via scaling events.

Not really sure what you mean by this. For the record, it is not entire nodes that are configured to use mirroring, but individual queues. In a RabbitMQ cluster, all schema definitions (e.g., queues, exchanges, bindings, users, etc) are replicated atomically between all nodes. Messages (inside queues) are normally not shared, but mirroring allows these to be replicated as well, for those queues to which the ha policy is applied.

> If a node dies, I can see via `rabbitmqctl cluster_status` that the node is still part of the cluster, but not a 'running node'.  What are the implications of this?

Client previously connected to that node will have been disconnected. If the master queue process for a mirrored queue previously resided on the now disconnected node, one of the remaining slaves - the most up date, i.e., complete replica - will be elected as the new master. When the node eventually re-establishes connectivity with the rest of the cluster, its copy of all the cluster-wide definitions (mentioned above) will be synchronised to reflect any changes that occurred whilst it was offline. If the node is selected (by an ha policy) to host mirrors for any HA/Mirrored queues, then slaves (i.e., replica queue processes) will be started for them.

>  If a client requests a message from a queue via an exchange, will the cluster service the request from an available node, knowing not to use the dead node?

That's not quite right - clients *publish* to exchanges and *consume* from queues. It sounds like your question is about location transparency (from the client's perspective) within a cluster however - in which case yes, that is right. Clients can connect to any node and see exchanges and queues that were declared on other nodes. Publishing and consuming works transparently, so that you can connect to any node in the cluster and publish to an exchange or consume from a queue, regardless of whether they were declared on the node to which you connected or not.

Queue will not however, magically "fail-over" if the node on which they were declared crashes - they will disappear from the cluster and any undelivered messages will be lost. If an exchange is bound to a queue on another node (and is not bound to any others!) and that node goes down, then the message will be lost since there is no longer a binding and the broker will drop the message in that case.

If a queue is mirrored (due to ha policy) however, you don't have to worry about that because a replica will take over and the message will be received there instead.

> Do I need to worry about cluster nodes versus running node; will the cluster remove a node that hasn't been running for some time, or should I programmatically clean up dead nodes?
> 

The cluster won't remove dead nodes. The assumption is that they will come back at some point. If they don't, then you are indeed responsible for removing them from the cluster.

> Also, how do clients connect to a cluster?  Do I put a load balancer in front of a cluster and have an application connect to the load balancer?

A lot of users seem to get along fine doing that.

> Or should an application connect to a particular cluster node, or any cluster node?  Apologies for the questions, but I'm not developing the app, just providing the infrastructure / middleware; not a fan of doing so without having a better understanding.
> 
> Any help, gratefully received!
> 

Sure not a problem. Hopefully I've answered some of the above without adding too much confusion! :)

Do spend some time reading the copious documentation on the website, and feel free to drop back here with any further questions.

Kind regards,

Tim