So your publishers connect to cluster A, which shovels a VIP that initially points to cluster B, and your consumers connect to B. �To upgrade B, you point the shovel VIP to new cluster C and start new consumers on C, then kill off the consumers on B when they've consumed all the messages. �Right? �How do you upgrade cluster A?<div>
<br></div><div>We'd like to have a solution that doesn't require our publishers/consumers to be touched/restarted at all; since we're starting with a clean sheet we can build whatever intelligence we need into our publishers/consumers. �One idea is to have 2 clusters, a primary A and secondary B, have publishers normally publish to A only, and have consumers consume from both A and B. �Then to upgrade A, we would need a way to trigger publishers to start publishing to B, and then once all the queues on A were drained by consumers we'd shut it down and upgrade. �Then we'd need a similar trigger to switch the publishers back to A.</div>
<div><br></div><div>I think if we define a publishing VIP in our load balancer and have publishers connect to that, we could accomplish the publishing redirection from A to B and back. �To switch them to cluster B, we would terminate those VIP connections and point the VIP to cluster B, and then vice-versa after A is upgraded. �We'd just need to make sure the publishers automatically reconnect to the VIP after being disconnected.</div>
<div><br></div><div>Sound right?</div><div><br></div><div>thx,</div><div>Chris</div><div><br><div class="gmail_quote">On Wed, Dec 5, 2012 at 7:19 PM, McIntosh Jason <span dir="ltr"><<a href="mailto:mcintoshj@gmail.com" target="_blank">mcintoshj@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Design we use:<br>
<br>
Shovel to a virtual ip rabbit cluster.<br>
Rabbit consumers on that cluster.<br>
Start new cluster.<br>
Can point consumers to hard coded cluster.<br>
Shift shovel and/or change VIP to point to new cluster.<br>
When all messages on old consumers cleaned, shut them down point em to new cluster.<br>
<br>
Our consumers are java apps built into rpms we deploy, so updates are just a restart/new rpm deploy<br>
<br>
This is pretty rough but hope it helps!<br>
Jason<br>
<br>
Sent from my iPhone<br>
<div class="HOEnZb"><div class="h5"><br>
On Dec 5, 2012, at 9:00 PM, Chris Toomey <<a href="mailto:ctoomey@gmail.com">ctoomey@gmail.com</a>> wrote:<br>
<br>
> Belated follow-up on this thread. �Can you talk more about how you manage<br>
> this? �E.g., how do you ensure that clients continue consuming from the old<br>
> cluster until all the messages published to it have been consumed? �And<br>
> during the transition time, consuming clients consume from both old and new<br>
> cluster, right? �What hardware/software do you use to gradually shift<br>
> publishing from old to new cluster?<br>
><br>
> We're just now designing and implementing our infrastructure and apps so<br>
> would like to get it as right as possible from the beginning. �Anyone else<br>
> using/recommend other approaches to this problem?<br>
><br>
> thx,<br>
> Chris<br>
><br>
><br>
> Laing, Michael P. wrote<br>
>> We bring up parallel infrastructure with a complete new cluster and<br>
>> gradually shift load to it using weighted routing.<br>
>><br>
>> This won't work for everybody, but we have designed our apps with this in<br>
>> mind.<br>
>><br>
>> Michael<br>
>><br>
>> From: Chris Toomey &lt;<br>
><br>
>> ctoomey@<br>
><br>
>> &lt;mailto:<br>
><br>
>> ctoomey@<br>
><br>
>> &gt;><br>
>> ...<br>
>> That's unfortunate about having to shut down the whole cluster to upgrade<br>
>> it -- it means that our applications will need to have some additional HA<br>
>> queueing mechanism upstream to buffer up the messages to be published<br>
>> during the downtime :-(.<br>
>><br>
>> What kinds of solutions are people using for that problem?<br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> View this message in context: <a href="http://rabbitmq.1065348.n5.nabble.com/High-availability-questions-tp23690p23889.html" target="_blank">http://rabbitmq.1065348.n5.nabble.com/High-availability-questions-tp23690p23889.html</a><br>
> Sent from the RabbitMQ mailing list archive at Nabble.com.<br>
> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div>