[rabbitmq-discuss] Fwd: Fwd: active/active

Thu May 13 20:02:22 BST 2010

Hi,

I'd like to make a brief comment on this.

Right now, we are starting to think properly about HA cases that
require something more refined than the active/passive set-up
described here: http://www.rabbitmq.com/pacemaker.html

There are a lot of these, depending on how you mix durability,
visibility, latency and domains of control.  Some of the NoSQL folks,
eg the Riak team, are working on related problems which we try to
study when we can.  We are actively working with customers who need
any kind of HA Rabbit, mostly behind the firewall but also in the
cloud.  We are also working on some 'cloud services' as I stated here:
http://skillsmatter.com/podcast/cloud-grid/use-cases-for-cloud-messaging

My point is - this is a useful discussion and we really appreciate any
and all feedback on this :-)

alexis

On Tue, May 11, 2010 at 7:59 PM, John Apps <johndapps at gmail.com> wrote:
> Jason suggested I post my response to his note rather than a private reply
> to him. Well, it just so happens that that is what I intended, but managed
> to get the email addresses all mixed - again.
> It is my belief that the world does need truly durable/persistent/reliable
> messaging. It is also the case that there are many applications out there
> which use it even when they had no need originally. DECmessageQ (Oracle
> Message Queue these days) is an example in point.
> ---------- Forwarded message ----------
> From: Jason J. W. Williams <jasonjwwilliams at gmail.com>
> Date: Tue, May 11, 2010 at 20:46
> Subject: Re: [rabbitmq-discuss] Fwd: active/active
> To: John Apps <johndapps at gmail.com>
>
>
> Hi John,
>
> It'd be great if you could share that with the list. Regarding
> duplication, I think that's easily addressed. If you're in a situation
> where the sender is going to re-send if they don't get confirmation it
> was accepted/processed on the other side, then you don't need durable
> messages. You're providing the durability.
>
> So the replay concept would only apply to durable messages and would
> allow all involved to make the assumption the message will eventually
> arrive and trust RabbitMQ to get it there...something that can't be
> assumed at the moment with clustering.
>
> -J
>
> On Tue, May 11, 2010 at 1:25 AM, John Apps <johndapps at gmail.com> wrote:
>> Option 1 below is neat and something that a product called Reliable
>> Transaction Router (RTR) has offered for many years with considerable
>> success. So, what, you might ask, has that to do with RabbitMQ? Well,
>> quite
>> simply, only to show that the concept is a viable one and has been used in
>> practice for many years.
>> One of the caveats with it is when the queues are feeding into two
>> databases, one a copy of the other; no, not mirrored in the classic sense,
>> rather, the messages are sent to both. The thing to take care of here is
>> that when messages are being replayed, i.e., one server has been down and
>> comes back up, whereupon the other server then plays the missed messages
>> to
>> it, is that the DB on the server processing the messages MUST not have
>> been
>> changed whilst the server was down.
>> What would be 'nice to have' in this scenario is a flag stating that the
>> messages are being re-played. This helps to avoid the problem of possible
>> duplicate messages by making one aware of the fact that a re-play is in
>> progress.
>> With the purchase of GemStone, perhaps the folks at RabbitMQ can use some
>> of
>> the distributed messaging technology from that product suite?
>>
>> On Tue, May 11, 2010 at 00:43, Jason J. W. Williams
>> <jasonjwwilliams at gmail.com> wrote:
>>>
>>> It was suggested to me I relay a convo I had with Alexis earlier this
>>> year making the case for better active/active fail-over in RabbitMQ.
>>> Particularly, when working with durable messaging.
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Jason J. W. Williams <jasonjwwilliams at gmail.com>
>>> Date: Mon, Jan 18, 2010 at 7:07 PM
>>> Subject: Re: active/active
>>> To: Alexis Richardson <alexis at rabbitmq.com>
>>>
>>>
>>> Hey Alexis,
>>>
>>> Currently, we have a single RMQ 1.5 pair that are setup
>>> active/passive. That is to say, our apps publish to node A and if it
>>> is unavailable publish to node B. Nodes A and B are standalone in that
>>> they are not in a cluster together. This is because the messages being
>>> published are durable, and in the event node A crashes any pending
>>> durable messages would not be auto-resubmitted if node A were
>>> clustered with node B (node B's queue essentially hides the pendings
>>> that were in the queue on node A before the crash).
>>>
>>> This set up works OK for us, as the load is fairly light and
>>> durability is more important for these tasks than availability.
>>>
>>> That being said we are designing a rework of our statistics logging
>>> infrastructure around Rabbit. Our stats is very high volume and
>>> availability is more important than durability (in fact none of the
>>> messages will be durable). So our intention is to create a new Rabbit
>>> cluster with three nodes and a load-balanced IP in front (since
>>> auto-redirect based on load is no longer supported....nasty memo to
>>> follow... ;) ).
>>>
>>> Our desire due to the fact that we're moving from a colo'd environment
>>> to a cloud environment, is to collapse both Rabbit "clusters"
>>> into a single cluster to maximize efficiency. It would also allow our
>>> existing app to benefit from the performance of a load-sharing
>>> environment. For that to happen however, Rabbit would have to do one
>>> of two things when clustered regarding durable messages:
>>>
>>> 1.) When a crashed node is restarted, replay the pending/durable
>>> messages into the queue on the node now responsible for the queue
>>> after the crash.
>>> 2.) Replicate all durable queues to more than one node, so that
>>> durable queue contents continue after a node crash.
>>>
>>> Frankly, option 1 is just fine with us and would solve our issue.
>>>
>>> Does this help? Thank you for being concerned.
>>>
>>> -J
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>>
>> --
>> ---
>> John Apps
>> (49) 171 869 1813
>>
>
>
>
> --
> ---
> John Apps
> (49) 171 869 1813
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>