[rabbitmq-discuss] Queue Replication over WAN

[Laing, Michael]

Hi Guillaume,

Wiser heads will no doubt correct me, but:

1. Durable queues != durable messages: for your messages to (possibly)
still be there after a crash you must indicate that they are persistent
(delivery_mode=2).

2. You can cross-link your exchanges/queues using shovels such that
messages appear in both servers in a symmetric manner. I have personally
found this easier to: understand, configure, implement, and operate than
federation, but don't let me discourage you from trying federation as well!

3. We do health checks and automated failover using AWS Route 53: a smart
DNS service - works fine, so you are good there.

4. Bonus: we actually exchange messages twice between servers (actually
clusters), once before they are 'processed' and once after they are
'processed'. We do this because our own processing steps are the least
reliable and most time consuming portion of each message pipeline, hence
failures tend to occur there. This results in duplicate messages after
processing (actually triplicate in our case). We have a 'resolver' layer
that de-duplicates and continuously reports on paths and latencies, as well
as the successful processing of each message.

Michael Laing
NYTimes


On Thu, Jul 11, 2013 at 9:19 PM, Guillaume Ouellet <gui.ouellet at gmail.com>wrote:

> Ok, I think I have found a way to solve this issue.
>
> My queues will be durable, so if the server crashes, the queues will be
> reloaded on boot up. The servers RAID configuration will make sure we don't
> loose any items from our queue.
>
> The producer client code will implement  a logic where if the primary
> hostname can't be reached, we will send the items to the secondary
> hostname. The consumers will listen to both hostname.
>
> 2 questions:
>
> 1. Is this approach flawed ?
> 2. I am using c# client, and I can't find in the doc any scheme for
> alternate nodes in case the primare fails? Is there any implementation
> similar to this, or I'll have to do my own ?
>
> Thanks
>
>
> On Wednesday, 10 July 2013 15:27:43 UTC-4, Guillaume Ouellet wrote:
>>
>> Hi all,
>>
>> I am trying to use RabbitMq so that I have one instance of RabbitMQ on
>> each of my 2 servers that are separated by a WAN. One server would be
>> active, the other "inactive", but the inactive would actually have the
>> queue from the active replicated real-time. This way, if the active server
>> goes down, the inactive server kicks in, and continues where the active
>> left off.
>>
>> It seemed like the clustering option does exactly this, but as I said
>> previously, I am over a WAN, and a WAN configuration is not recommended.
>>
>> My question is how would I implement the previous logic via shovels or
>> federations ? I can't seem to get my head around this looking at the
>> documentation.
>>
>> Thank you
>>
>>
>>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130711/b668effb/attachment.htm>