[rabbitmq-discuss] High availability questions

[Chris Toomey]

So your publishers connect to cluster A, which shovels a VIP that initially
points to cluster B, and your consumers connect to B.  To upgrade B, you
point the shovel VIP to new cluster C and start new consumers on C, then
kill off the consumers on B when they've consumed all the messages.  Right?
 How do you upgrade cluster A?

We'd like to have a solution that doesn't require our publishers/consumers
to be touched/restarted at all; since we're starting with a clean sheet we
can build whatever intelligence we need into our publishers/consumers.  One
idea is to have 2 clusters, a primary A and secondary B, have publishers
normally publish to A only, and have consumers consume from both A and B.
 Then to upgrade A, we would need a way to trigger publishers to start
publishing to B, and then once all the queues on A were drained by
consumers we'd shut it down and upgrade.  Then we'd need a similar trigger
to switch the publishers back to A.

I think if we define a publishing VIP in our load balancer and have
publishers connect to that, we could accomplish the publishing redirection
from A to B and back.  To switch them to cluster B, we would terminate
those VIP connections and point the VIP to cluster B, and then vice-versa
after A is upgraded.  We'd just need to make sure the publishers
automatically reconnect to the VIP after being disconnected.

Sound right?

thx,
Chris

On Wed, Dec 5, 2012 at 7:19 PM, McIntosh Jason <mcintoshj at gmail.com> wrote:

> Design we use:
>
> Shovel to a virtual ip rabbit cluster.
> Rabbit consumers on that cluster.
> Start new cluster.
> Can point consumers to hard coded cluster.
> Shift shovel and/or change VIP to point to new cluster.
> When all messages on old consumers cleaned, shut them down point em to new
> cluster.
>
> Our consumers are java apps built into rpms we deploy, so updates are just
> a restart/new rpm deploy
>
> This is pretty rough but hope it helps!
> Jason
>
> Sent from my iPhone
>
> On Dec 5, 2012, at 9:00 PM, Chris Toomey <ctoomey at gmail.com> wrote:
>
> > Belated follow-up on this thread.  Can you talk more about how you manage
> > this?  E.g., how do you ensure that clients continue consuming from the
> old
> > cluster until all the messages published to it have been consumed?  And
> > during the transition time, consuming clients consume from both old and
> new
> > cluster, right?  What hardware/software do you use to gradually shift
> > publishing from old to new cluster?
> >
> > We're just now designing and implementing our infrastructure and apps so
> > would like to get it as right as possible from the beginning.  Anyone
> else
> > using/recommend other approaches to this problem?
> >
> > thx,
> > Chris
> >
> >
> > Laing, Michael P. wrote
> >> We bring up parallel infrastructure with a complete new cluster and
> >> gradually shift load to it using weighted routing.
> >>
> >> This won't work for everybody, but we have designed our apps with this
> in
> >> mind.
> >>
> >> Michael
> >>
> >> From: Chris Toomey &lt;
> >
> >> ctoomey@
> >
> >> &lt;mailto:
> >
> >> ctoomey@
> >
> >> &gt;>
> >> ...
> >> That's unfortunate about having to shut down the whole cluster to
> upgrade
> >> it -- it means that our applications will need to have some additional
> HA
> >> queueing mechanism upstream to buffer up the messages to be published
> >> during the downtime :-(.
> >>
> >> What kinds of solutions are people using for that problem?
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://rabbitmq.1065348.n5.nabble.com/High-availability-questions-tp23690p23889.html
> > Sent from the RabbitMQ mailing list archive at Nabble.com.
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121206/884b8135/attachment.htm>