[rabbitmq-discuss] Clustering and scaling

[Stuart Munro]

Hi Eugene,

-- Tell more about the durability guarantees that you expect.
-- You can use active/active HA, but I think it imposes a performance
penalty.
-- Do you expect the dead server to come back online sometime (after a
-- restart), or are the failures permanent?

If the server dies, we will rebuild a new one and add it back in.
If I use active/active what happens if two queue consumers hit the
RabbitMQ cluster at the same time and one goes to server A and the other
goes to server B? Will they not get the same job? How do I guarantee the
queue is consistent between multiple RabbitMQ's

-- You can just make all consumers listen on all the servers at once, or
-- partition them between the consumers (e.g. each consumer listens to a
-- random server).


But if I put a load balancer between the queue consumer servers and the
RabbitMQ servers, the queue consumer servers would only need to know the
load balancer address - therefore enabling movement of the RabbitMQ
servers.


Thanks again,

Stuart







On 08/09/2011 13:08, "Eugene Kirpichov" <ekirpichov at gmail.com> wrote:

>Hi,
>
>On Thu, Sep 8, 2011 at 3:42 PM, Stuart Munro <stuart at state.it> wrote:
>> Sorry Eugene, my understanding may be wrong, I'll try again.
>>
>> 1. Ideally I would like a few queue consumers servers running with the
>> ability to add more if/when needed on-the-fly (this doesn't affect
>> RabbitMQ, but I'll get to that in a second)
>>
>> 2. I would like multiple RabbitMQ servers that are redundant, if one of
>> the RabbitMQ servers dies the cluster should continue working as normal,
>Tell more about the durability guarantees that you expect.
>You can use active/active HA, but I think it imposes a performance
>penalty.
>Do you expect the dead server to come back online sometime (after a
>restart), or are the failures permanent?
>
>> I
>> also want the ability to add additional RabbitMQ servers on-the-fly
>> without having to tell the queue consumers that a new RabbitMQ server
>>now
>> exists. This is the bit I don't quite understand how to implement.
>
>There's no out-of-the-box provisioning for this at all. You'll have to
>add your own infrastructure to support this.
>
>>
>>
>> In order for section 2 to work (from my understanding) I would need to
>>put
>> a load balancer in front of the RabbitMQ servers, therefore the queue
>> consumers don't care about what server they pick-up the job from - it
>>also
>> allows me to add RabbitMQ servers on-the-fly without reconfiguring as
>>the
>> queue consumer would simply hit the load balancer.
>You can just make all consumers listen on all the servers at once, or
>partition them between the consumers (e.g. each consumer listens to a
>random server).
>
>>
>> How do people currently implement multiple RabbitMQ servers?
>>
>> Thanks
>>
>>
>> --
>> Stuart Munro
>>
>> e: stuart at state.it
>> w: www.state.it
>> m: +44 (0) 7738 755574
>>
>>
>>
>>
>> On 07/09/2011 19:04, "Eugene Kirpichov" <ekirpichov at gmail.com> wrote:
>>
>>>Hi Stuart,
>>>
>>>I haven't fully understood whether you want scaling for redundancy or
>>>scaling for load balancing.
>>>In the first case I don't see why you're interested in the number of
>>>consumers.
>>>In the second case I don't see why you're interested in replication.
>>>
>>>On Wed, Sep 7, 2011 at 7:49 PM, Stuart Munro <stuart at state.it> wrote:
>>>> Hi,
>>>> I was wondering someone could give me a few pointers about using
>>>>RabbitMQ in
>>>> cluster mode.
>>>> So far we have 2 RabbitMQ's in cluster mode (rb-1 and rb-2), rb-2 is
>>>>in
>>>>RAM
>>>> mode.
>>>> How do other people setup redundancy in terms of queue consumers
>>>>hitting the
>>>> cluster?
>>>> So, for example could I have a load balancer sitting in front of the 2
>>>>rb's
>>>> and in a round-robin approach on the load balancer distribute load
>>>>between
>>>> them ­ this raises the question ­ how quick is the replication between
>>>>the
>>>> nodes?
>>>> Alternatively, do I setup a failover implementation using something
>>>>like
>>>> HAProxy? But this raises the question how to I scale horizontally?
>>>> Thank you very much for your time, much appreciated.
>>>> Stuart
>>>>
>>>> _______________________________________________
>>>> rabbitmq-discuss mailing list
>>>> rabbitmq-discuss at lists.rabbitmq.com
>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>>
>>>>
>>>
>>>
>>>
>>>--
>>>Eugene Kirpichov
>>>Principal Engineer, Mirantis Inc. http://www.mirantis.com/
>>>Editor, http://fprog.ru/
>>
>>
>
>
>
>-- 
>Eugene Kirpichov
>Principal Engineer, Mirantis Inc. http://www.mirantis.com/
>Editor, http://fprog.ru/