[rabbitmq-discuss] High availability questions
ctoomey at gmail.com
Thu Dec 6 19:06:12 GMT 2012
So your publishers connect to cluster A, which shovels a VIP that initially
points to cluster B, and your consumers connect to B. To upgrade B, you
point the shovel VIP to new cluster C and start new consumers on C, then
kill off the consumers on B when they've consumed all the messages. Right?
How do you upgrade cluster A?
We'd like to have a solution that doesn't require our publishers/consumers
to be touched/restarted at all; since we're starting with a clean sheet we
can build whatever intelligence we need into our publishers/consumers. One
idea is to have 2 clusters, a primary A and secondary B, have publishers
normally publish to A only, and have consumers consume from both A and B.
Then to upgrade A, we would need a way to trigger publishers to start
publishing to B, and then once all the queues on A were drained by
consumers we'd shut it down and upgrade. Then we'd need a similar trigger
to switch the publishers back to A.
I think if we define a publishing VIP in our load balancer and have
publishers connect to that, we could accomplish the publishing redirection
from A to B and back. To switch them to cluster B, we would terminate
those VIP connections and point the VIP to cluster B, and then vice-versa
after A is upgraded. We'd just need to make sure the publishers
automatically reconnect to the VIP after being disconnected.
On Wed, Dec 5, 2012 at 7:19 PM, McIntosh Jason <mcintoshj at gmail.com> wrote:
> Design we use:
> Shovel to a virtual ip rabbit cluster.
> Rabbit consumers on that cluster.
> Start new cluster.
> Can point consumers to hard coded cluster.
> Shift shovel and/or change VIP to point to new cluster.
> When all messages on old consumers cleaned, shut them down point em to new
> Our consumers are java apps built into rpms we deploy, so updates are just
> a restart/new rpm deploy
> This is pretty rough but hope it helps!
> Sent from my iPhone
> On Dec 5, 2012, at 9:00 PM, Chris Toomey <ctoomey at gmail.com> wrote:
> > Belated follow-up on this thread. Can you talk more about how you manage
> > this? E.g., how do you ensure that clients continue consuming from the
> > cluster until all the messages published to it have been consumed? And
> > during the transition time, consuming clients consume from both old and
> > cluster, right? What hardware/software do you use to gradually shift
> > publishing from old to new cluster?
> > We're just now designing and implementing our infrastructure and apps so
> > would like to get it as right as possible from the beginning. Anyone
> > using/recommend other approaches to this problem?
> > thx,
> > Chris
> > Laing, Michael P. wrote
> >> We bring up parallel infrastructure with a complete new cluster and
> >> gradually shift load to it using weighted routing.
> >> This won't work for everybody, but we have designed our apps with this
> >> mind.
> >> Michael
> >> From: Chris Toomey <
> >> ctoomey@
> >> <mailto:
> >> ctoomey@
> >> >>
> >> ...
> >> That's unfortunate about having to shut down the whole cluster to
> >> it -- it means that our applications will need to have some additional
> >> queueing mechanism upstream to buffer up the messages to be published
> >> during the downtime :-(.
> >> What kinds of solutions are people using for that problem?
> > --
> > View this message in context:
> > Sent from the RabbitMQ mailing list archive at Nabble.com.
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss