[rabbitmq-discuss] Clustering and scaling

[Eugene Kirpichov]

Hi,

On Thu, Sep 8, 2011 at 3:42 PM, Stuart Munro <stuart at state.it> wrote:
> Sorry Eugene, my understanding may be wrong, I'll try again.
>
> 1. Ideally I would like a few queue consumers servers running with the
> ability to add more if/when needed on-the-fly (this doesn't affect
> RabbitMQ, but I'll get to that in a second)
>
> 2. I would like multiple RabbitMQ servers that are redundant, if one of
> the RabbitMQ servers dies the cluster should continue working as normal,
Tell more about the durability guarantees that you expect.
You can use active/active HA, but I think it imposes a performance penalty.
Do you expect the dead server to come back online sometime (after a
restart), or are the failures permanent?

> I
> also want the ability to add additional RabbitMQ servers on-the-fly
> without having to tell the queue consumers that a new RabbitMQ server now
> exists. This is the bit I don't quite understand how to implement.

There's no out-of-the-box provisioning for this at all. You'll have to
add your own infrastructure to support this.

>
>
> In order for section 2 to work (from my understanding) I would need to put
> a load balancer in front of the RabbitMQ servers, therefore the queue
> consumers don't care about what server they pick-up the job from - it also
> allows me to add RabbitMQ servers on-the-fly without reconfiguring as the
> queue consumer would simply hit the load balancer.
You can just make all consumers listen on all the servers at once, or
partition them between the consumers (e.g. each consumer listens to a
random server).

>
> How do people currently implement multiple RabbitMQ servers?
>
> Thanks
>
>
> --
> Stuart Munro
>
> e: stuart at state.it
> w: www.state.it
> m: +44 (0) 7738 755574
>
>
>
>
> On 07/09/2011 19:04, "Eugene Kirpichov" <ekirpichov at gmail.com> wrote:
>
>>Hi Stuart,
>>
>>I haven't fully understood whether you want scaling for redundancy or
>>scaling for load balancing.
>>In the first case I don't see why you're interested in the number of
>>consumers.
>>In the second case I don't see why you're interested in replication.
>>
>>On Wed, Sep 7, 2011 at 7:49 PM, Stuart Munro <stuart at state.it> wrote:
>>> Hi,
>>> I was wondering someone could give me a few pointers about using
>>>RabbitMQ in
>>> cluster mode.
>>> So far we have 2 RabbitMQ's in cluster mode (rb-1 and rb-2), rb-2 is in
>>>RAM
>>> mode.
>>> How do other people setup redundancy in terms of queue consumers
>>>hitting the
>>> cluster?
>>> So, for example could I have a load balancer sitting in front of the 2
>>>rb's
>>> and in a round-robin approach on the load balancer distribute load
>>>between
>>> them ­ this raises the question ­ how quick is the replication between
>>>the
>>> nodes?
>>> Alternatively, do I setup a failover implementation using something like
>>> HAProxy? But this raises the question how to I scale horizontally?
>>> Thank you very much for your time, much appreciated.
>>> Stuart
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>
>>
>>
>>--
>>Eugene Kirpichov
>>Principal Engineer, Mirantis Inc. http://www.mirantis.com/
>>Editor, http://fprog.ru/
>
>



-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/