[rabbitmq-discuss] Clustering and scaling

Thu Sep 8 13:35:15 BST 2011

Hi,

2011/9/8 Stuart Munro <stuart at state.it>:
>
> Hi Eugene,
>
> -- Tell more about the durability guarantees that you expect.
> -- You can use active/active HA, but I think it imposes a performance
> penalty.
> -- Do you expect the dead server to come back online sometime (after a
> -- restart), or are the failures permanent?
>
> If the server dies, we will rebuild a new one and add it back in.
> If I use active/active what happens if two queue consumers hit the
> RabbitMQ cluster at the same time and one goes to server A and the other
> goes to server B? Will they not get the same job? How do I guarantee the
> queue is consistent between multiple RabbitMQ's
I don't know. I hope someone else on the list can tell more about
active/active HA.

>
> -- You can just make all consumers listen on all the servers at once, or
> -- partition them between the consumers (e.g. each consumer listens to a
> -- random server).
>
>
> But if I put a load balancer between the queue consumer servers and the
> RabbitMQ servers, the queue consumer servers would only need to know the
> load balancer address - therefore enabling movement of the RabbitMQ
> servers.
Yes, this could be an option, though of course the load balancer
itself would have to be able to sustain the required load.
I am not aware of an existing load balancing solution for RabbitMQ,
though there might be one.

>
>
> Thanks again,
>
> Stuart
>
>
>
>
>
>
>
> On 08/09/2011 13:08, "Eugene Kirpichov" <ekirpichov at gmail.com> wrote:
>
>>Hi,
>>
>>On Thu, Sep 8, 2011 at 3:42 PM, Stuart Munro <stuart at state.it> wrote:
>>> Sorry Eugene, my understanding may be wrong, I'll try again.
>>>
>>> 1. Ideally I would like a few queue consumers servers running with the
>>> ability to add more if/when needed on-the-fly (this doesn't affect
>>> RabbitMQ, but I'll get to that in a second)
>>>
>>> 2. I would like multiple RabbitMQ servers that are redundant, if one of
>>> the RabbitMQ servers dies the cluster should continue working as normal,
>>Tell more about the durability guarantees that you expect.
>>You can use active/active HA, but I think it imposes a performance
>>penalty.
>>Do you expect the dead server to come back online sometime (after a
>>restart), or are the failures permanent?
>>
>>> I
>>> also want the ability to add additional RabbitMQ servers on-the-fly
>>> without having to tell the queue consumers that a new RabbitMQ server
>>>now
>>> exists. This is the bit I don't quite understand how to implement.
>>
>>There's no out-of-the-box provisioning for this at all. You'll have to
>>add your own infrastructure to support this.
>>
>>>
>>>
>>> In order for section 2 to work (from my understanding) I would need to
>>>put
>>> a load balancer in front of the RabbitMQ servers, therefore the queue
>>> consumers don't care about what server they pick-up the job from - it
>>>also
>>> allows me to add RabbitMQ servers on-the-fly without reconfiguring as
>>>the
>>> queue consumer would simply hit the load balancer.
>>You can just make all consumers listen on all the servers at once, or
>>partition them between the consumers (e.g. each consumer listens to a
>>random server).
>>
>>>
>>> How do people currently implement multiple RabbitMQ servers?
>>>
>>> Thanks
>>>
>>>
>>> --
>>> Stuart Munro
>>>
>>> e: stuart at state.it
>>> w: www.state.it
>>> m: +44 (0) 7738 755574
>>>
>>>
>>>
>>>
>>> On 07/09/2011 19:04, "Eugene Kirpichov" <ekirpichov at gmail.com> wrote:
>>>
>>>>Hi Stuart,
>>>>
>>>>I haven't fully understood whether you want scaling for redundancy or
>>>>scaling for load balancing.
>>>>In the first case I don't see why you're interested in the number of
>>>>consumers.
>>>>In the second case I don't see why you're interested in replication.
>>>>
>>>>On Wed, Sep 7, 2011 at 7:49 PM, Stuart Munro <stuart at state.it> wrote:
>>>>> Hi,
>>>>> I was wondering someone could give me a few pointers about using
>>>>>RabbitMQ in
>>>>> cluster mode.
>>>>> So far we have 2 RabbitMQ's in cluster mode (rb-1 and rb-2), rb-2 is
>>>>>in
>>>>>RAM
>>>>> mode.
>>>>> How do other people setup redundancy in terms of queue consumers
>>>>>hitting the
>>>>> cluster?
>>>>> So, for example could I have a load balancer sitting in front of the 2
>>>>>rb's
>>>>> and in a round-robin approach on the load balancer distribute load
>>>>>between
>>>>> them  this raises the question  how quick is the replication between
>>>>>the
>>>>> nodes?
>>>>> Alternatively, do I setup a failover implementation using something
>>>>>like
>>>>> HAProxy? But this raises the question how to I scale horizontally?
>>>>> Thank you very much for your time, much appreciated.
>>>>> Stuart
>>>>>
>>>>> _______________________________________________
>>>>> rabbitmq-discuss mailing list
>>>>> rabbitmq-discuss at lists.rabbitmq.com
>>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>--
>>>>Eugene Kirpichov
>>>>Principal Engineer, Mirantis Inc. http://www.mirantis.com/
>>>>Editor, http://fprog.ru/
>>>
>>>
>>
>>
>>
>>--
>>Eugene Kirpichov
>>Principal Engineer, Mirantis Inc. http://www.mirantis.com/
>>Editor, http://fprog.ru/
>
>


-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/