[rabbitmq-discuss] High Scalability and High Availability...How to?!!

[Jason McIntosh]

Paul:

You'll get load balanced connections this way to the cluster.  But queues
are 'mastered' on a single node in the cluster, and mirrored to other
nodes.  You can control somewhat which node the queues are mastered on.  If
you're declaring the queues in your publisher/consumer, then the queues are
mastered I believe on the connected node, so load balancing would get you
there.  For mirroring though, you can say build a 5 node cluster, mirror
your queue to 3 nodes, and then try and spread your queues out across the
nodes in such a way that allows growth.  You have redundancy x3, and the
ability to grow performance wise.  The biggest trick here is that your apps
may connect to a node that the queue isn't mastered on but it seems the
rabbit server (assuming it's not a master) acts as a proxy to the mastered
queue.  And you get pretty solid scalability/performance with such an
architecture.  At least, we have :)

Jason


On Fri, Jun 13, 2014 at 2:40 AM, Paul Holyoake <paul.holyoake at googlemail.com
> wrote:

> I've recently had a similar HA and scalability requirement. I couldn't
> find a perfect answer. I achieved HA through a network loadbalancer across
> a pair of clustered RabbitMQ nodes, with mirrored queues. If the NLB fails
> to connect on port 5672 of the primary it failsover to the secondary node.
> All queue data is available on this node due to the queue mirroring.
>
> In order to achieve scalability my assumption is to logically partition
> the data messages and have over clusters to meet that requirement.
>
> Would the NLB in round robin mode be viable and have the nodes effectively
> active/active?
>
>
>
> On Thursday, June 12, 2014 5:47:30 PM UTC+1, Shadi Mari wrote:
>
>> Hello,
>>
>> I am new to Rabbitmq, however i have built so many messaging solution on
>> top of other stacks..
>>
>> I am trying to leverage the capabilities of RabbitMQ in one of my
>> solutions where HA and high scalability  is a must; so far the better
>> option the came out of my research
>> is to use active/active cluster and mirrored queues in order to provide
>> HA, otherwise using active/standby where the queue lives on the node it was
>> created on, will not achieve HA requirement.
>>
>> On the other hand, this should be mixed a load balancer, where consumers
>> and publishers are connected to the cluster through in order to direct
>> consumers/publishers to a different node each time the connection takes
>> place.
>>
>> However, my understanding says that RabbiqMQ in the case of mirrored
>> queue will connect consumers and publishers to the master node which will
>> not help in distributing load across all nodes. I am not sure but it came
>> to my knowledge that SoundCloud uses the same approach!
>>
>> Is my understanding above correct? if Yes, then how do companies that
>> uses RabbitMQ and require HA and high scalability work it out.
>>
>> Thank you
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>


-- 
Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140613/3021fc1b/attachment.html>