[rabbitmq-discuss] Clustering and scaling
stuart at state.it
Thu Sep 8 13:26:06 BST 2011
-- Tell more about the durability guarantees that you expect.
-- You can use active/active HA, but I think it imposes a performance
-- Do you expect the dead server to come back online sometime (after a
-- restart), or are the failures permanent?
If the server dies, we will rebuild a new one and add it back in.
If I use active/active what happens if two queue consumers hit the
RabbitMQ cluster at the same time and one goes to server A and the other
goes to server B? Will they not get the same job? How do I guarantee the
queue is consistent between multiple RabbitMQ's
-- You can just make all consumers listen on all the servers at once, or
-- partition them between the consumers (e.g. each consumer listens to a
-- random server).
But if I put a load balancer between the queue consumer servers and the
RabbitMQ servers, the queue consumer servers would only need to know the
load balancer address - therefore enabling movement of the RabbitMQ
On 08/09/2011 13:08, "Eugene Kirpichov" <ekirpichov at gmail.com> wrote:
>On Thu, Sep 8, 2011 at 3:42 PM, Stuart Munro <stuart at state.it> wrote:
>> Sorry Eugene, my understanding may be wrong, I'll try again.
>> 1. Ideally I would like a few queue consumers servers running with the
>> ability to add more if/when needed on-the-fly (this doesn't affect
>> RabbitMQ, but I'll get to that in a second)
>> 2. I would like multiple RabbitMQ servers that are redundant, if one of
>> the RabbitMQ servers dies the cluster should continue working as normal,
>Tell more about the durability guarantees that you expect.
>You can use active/active HA, but I think it imposes a performance
>Do you expect the dead server to come back online sometime (after a
>restart), or are the failures permanent?
>> also want the ability to add additional RabbitMQ servers on-the-fly
>> without having to tell the queue consumers that a new RabbitMQ server
>> exists. This is the bit I don't quite understand how to implement.
>There's no out-of-the-box provisioning for this at all. You'll have to
>add your own infrastructure to support this.
>> In order for section 2 to work (from my understanding) I would need to
>> a load balancer in front of the RabbitMQ servers, therefore the queue
>> consumers don't care about what server they pick-up the job from - it
>> allows me to add RabbitMQ servers on-the-fly without reconfiguring as
>> queue consumer would simply hit the load balancer.
>You can just make all consumers listen on all the servers at once, or
>partition them between the consumers (e.g. each consumer listens to a
>> How do people currently implement multiple RabbitMQ servers?
>> Stuart Munro
>> e: stuart at state.it
>> w: www.state.it
>> m: +44 (0) 7738 755574
>> On 07/09/2011 19:04, "Eugene Kirpichov" <ekirpichov at gmail.com> wrote:
>>>I haven't fully understood whether you want scaling for redundancy or
>>>scaling for load balancing.
>>>In the first case I don't see why you're interested in the number of
>>>In the second case I don't see why you're interested in replication.
>>>On Wed, Sep 7, 2011 at 7:49 PM, Stuart Munro <stuart at state.it> wrote:
>>>> I was wondering someone could give me a few pointers about using
>>>> cluster mode.
>>>> So far we have 2 RabbitMQ's in cluster mode (rb-1 and rb-2), rb-2 is
>>>> How do other people setup redundancy in terms of queue consumers
>>>> So, for example could I have a load balancer sitting in front of the 2
>>>> and in a round-robin approach on the load balancer distribute load
>>>> them this raises the question how quick is the replication between
>>>> Alternatively, do I setup a failover implementation using something
>>>> HAProxy? But this raises the question how to I scale horizontally?
>>>> Thank you very much for your time, much appreciated.
>>>> rabbitmq-discuss mailing list
>>>> rabbitmq-discuss at lists.rabbitmq.com
>>>Principal Engineer, Mirantis Inc. http://www.mirantis.com/
>Principal Engineer, Mirantis Inc. http://www.mirantis.com/
More information about the rabbitmq-discuss