[rabbitmq-discuss] Simultaneous Scalability and HA

Tim Watson tim at rabbitmq.com
Fri Jun 21 11:17:46 BST 2013


On 21 Jun 2013, at 09:39, Hudson Jiang wrote:
> However, for achieving HA, the suggested approach is the built-in active-active option for queues: mirrored queues. And it's recommended to use: 
> queue_args = {"x-ha-policy" : "all" }
> 

Where does it say the "all" policy is the recommend one? Look at it like this: mirror queues (i.e., HA in RabbitMQ) buys you resilience to node failure, which might increase availability. But it doesn't buy you anything in terms of "distributing load", since HA actually adds load (as a result of replication) to the system.

> Is above cluster approach for scalability conflicted with the mirrored queues approach for HA (do I misunderstand anything)? How can I achieve both of them simultaneously?
> 

Clustering provides a consistent view of all the (clustered) brokers from each node, which means you can connect to any node and interact with objects (i.e., publish to exchanges or consume from queues) on any node in the cluster.

Mirroring / HA provides replication of contents for 1 queue, that you've chosen to mirror, and you can maintain the queue's state across however many replicas you see fit by setting the appropriate policy. If you mirrored all queues across all nodes, you'd find that hurts performance. If you mirror no queues, you might loose data if one of your nodes crashes.

Now, to Michael's point: what do *you* want to do here? Scalability is an almost meaningless term used in isolation. Are you trying to achieve:

* higher levels of throughput than a single node can manage?
* higher levels of availability/resilience just in case one or more nodes crash?
* ability to add more capacity (e.g., # clients, volume of work/throughput, etc) to a live system?
* etc??

Note that the "A" in HA stands for "Availability" not "throughput" or "performance", though we think that acceptable levels of throughput and performance are achievable with mirror queues, considering the compromises necessary to guarantee consistency and avoid data loss.

Other options for utilising clusters include load balancing in front of the cluster, using federation or the shovel plugin instead of clustering to distribute messages between nodes (or data centres).

Cheers,
Tim


More information about the rabbitmq-discuss mailing list