[rabbitmq-discuss] Fwd: Client-connection failover workarounds (ruby)

[Peter Fitzgibbons]

HI Matthias,

Could you direct me to an example of how to set up a "stand alone" shovel?
The only example[1] i found is demonstrating an embedded shovel as rabbitmq
plugin.  This seems to be my degenerate case all over again.  No?

[1]
http://hopper.squarespace.com/blog/2008/6/22/introducing-shovel-an-amqp-relay.html

Peter Fitzgibbons
(847) 859-9550
Email: peter.fitzgibbons at gmail.com
IM GTalk: peter.fitzgibbons
IM AOL: peter.fitzgibbons at gmail.com


On Sat, Feb 6, 2010 at 12:42 PM, Matthias Radestock <matthias at lshift.net>wrote:

> Peter,
>
> Peter Fitzgibbons wrote:
>
>> I'm thinking about how to handle client-connection failover.  By this
>> I mean clientA has a producerP that needs to have reliable and
>> tolerant connections to rabbitA and rabbitB.  This is so that if the
>> connection/pid of rabbitA goes down, producerP sends it's next message
>> to rabbitB with as little handling (and wall-clock time) as possible.
>> [...]
>>
>> The best solution I can think of right now is to have a rabbitP on the
>> same machine as producerP, which is clustered to the "real" cluster,
>> so if rabbitA goes down, the built-in clustering failover will handle
>> the proper interaction.  My issue with this is considering how this is
>> configured and maintained when the farm of servers with producerP gets
>> to be 1000+.  Even if this doesn't scale to the google-farm level,
>> what about farm = 10, 30 ?
>>
>
> Having a local broker is indeed a common approach to get higher resilience,
> particularly since it allows producers to offload messages even when the
> connection to the main brokers is down for prolonged periods of time, e.g.
> in the event the main brokers are across a WAN and there is a WAN failure.
>
> As you suspect, making all these brokers part of a cluster would indeed be
> problematic for high node counts, even more so when going across a WAN.
> Besides, RabbitMQ's clustering is really designed for scaling, not improved
> reliability.
>
> But you do not need to cluster the brokers. Instead you could stick our
> experimental shovel (see
> http://www.lshift.net/blog/2010/02/01/rabbitmq-shovel-message-relocation-equipment)
> onto each local node and configure it to shovel messages to the main broker
> pair. The shovel has built-in re-connection and failover logic for that.
>
> Of course having a local broker is no panacea for reliability, since that
> broker itself becomes a point of failure. Whether that is a problem depends
> on how high you place the reliability bar.
>
> An alternative approachh is to make the clients themselves deal with
> re-connection and failover. Check out the experimental
> http://hg.rabbitmq.com/rabbitmq-java-messagepatterns/ and
> http://hg.rabbitmq.com/rabbitmq-dotnet-messagepatterns/ libraries which do
> that for a specific type of messaging (point-to-point, though most of the
> logic should be easily ported to other types).
>
>
> Regards,
>
> Matthias
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100208/341412d8/attachment.htm