[rabbitmq-discuss] Clustering over a WAN

[Chris Hampson]

Thanks Simon and Matthew for the quick responses.

I was beginning to figure that perhaps what we were trying to do was going to be problematic, which is a shame as it would've been a very neat solution to what we're trying to do.

In essence our use case is as follows:

We have groups of users and resources at each of the sites, each of these sites also have a compute cluster. We wanted a system whereby a user could essentially have a client that could subscribe to events in a site generated by the compute cluster, jobs starting/finishing, status updates, warnings etc. and act accordingly, be it automatically or through some kind of notification and then human interaction.

We also wanted to allow (very selectively) for a user at one site to subscribe for certain messages published to other sites (placing a ban on blanket subscriptions to minimise unnecessary bandwidth being wasted).

We figured this could be implemented with a cluster, all clients connecting to their local broker but publishing to a common exchange. In our tests this worked well, with users also being able to selectively get messages from other sites as well as their own (working out a scheme of users and permissions for this was fun, but we achieved it in the end).

This was essentially it, we then planned other applications on top of this base, it was shaping up nicely until we started experiencing the cluster break ups :-\

I've just looked into the shovel plugin further, and it looks as if it may not be quite dynamic enough for our use (we don't want to be re-tweaking config files every time we set up a new pattern) :-( it's beginning to look like the best way forward for now is separate brokers with custom clients acting as the shovels between sites, consuming on one, then publishing to another as necessary.

Any advice and/or tips are most welcome here, we're quite new to the world of messaging, and may be approaching this backwards (which would be a shame, as we thought we'd found a very nice solution). ;-)

Cheers,

Chris

> -----Original Message-----
> From: rabbitmq-discuss-bounces at lists.rabbitmq.com [mailto:rabbitmq-
> discuss-bounces at lists.rabbitmq.com] On Behalf Of Matthew Sackman
> Sent: 28 February 2011 15:51
> To: rabbitmq-discuss at lists.rabbitmq.com
> Subject: Re: [rabbitmq-discuss] Clustering over a WAN
>
> On Mon, Feb 28, 2011 at 03:27:42PM +0000, Chris Hampson wrote:
> > We're currently attempting to maintain a RabbitMQ cluster over a WAN
> between some of our sites, 2 in the US, one in the UK and another in
> India.
>
> That's a bad idea.
>
> > For the most part this seems to be working fine, but it seems a
> little fragile and we can't seem to get it to recover from failures
> very well.
>
> Correct. Mnesia, which Rabbit uses to store all sorts of data (though
> not messages), copes poorly with network partitions.
>
> > If anyone can provide any advice to aid our situation we'd be most
> > grateful (even if it is "don't do that you loony, separate them out
> > and shovel messages between sites when necessary")
>
> Yeah, the latter's a fair summary of my advice on this ;) We do state
> quite prominantly that clustering is not for HA, though I admit we
> state
> it on http://www.rabbitmq.com/pacemaker.html rather than
> http://www.rabbitmq.com/clustering.html which is not really as helpful
> as it could be.
>
> The problem is that mnesia, and by extension Rabbit, goes for the
> Consistency and Availability part of the CAP triangle, and so just does
> not cope well with partitions ocurring.
> http://en.wikipedia.org/wiki/CAP_theorem
>
> We are working on solving some of these problems in a number of
> different ways. If the shovel is a viable solution then I'd recommend
> you use that, otherwise please let us know some more details of your
> use
> case and we should be able to advise further.
>
> Best wishes,
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.