So our "publishers" publish to a local rabbitmq in a remote data center. Then each shovels their messages to a warehousing cluster for consumption. In our case, upgrading a remote system is just pulling that remote system out of the load balancer, waiting for all it's messages to shovel, stopping, upgrading it, then rinse & repeat. <div>
<br></div><div>In theory, you could do something along the lines of what you're talking about by clustering your publisher rabbit boxes together, just like you'd do for the consumer process. As long as your load balancer stays up reliably (which they're supposed to do). I HAVE had issues with connection resets breaking clients, so like anything else, make sure you test it. Connection resets though when shoveling seem to handle such a situation fine - I'm guessing shovel is designed to work even when it's connection breaks :) Which is AWESOME btw.<br>
<div><br></div><div>Jason</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 6, 2012 at 1:06 PM, Chris Toomey <span dir="ltr"><<a href="mailto:ctoomey@gmail.com" target="_blank">ctoomey@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">So your publishers connect to cluster A, which shovels a VIP that initially points to cluster B, and your consumers connect to B. To upgrade B, you point the shovel VIP to new cluster C and start new consumers on C, then kill off the consumers on B when they've consumed all the messages. Right? How do you upgrade cluster A?<div>
<br></div><div>We'd like to have a solution that doesn't require our publishers/consumers to be touched/restarted at all; since we're starting with a clean sheet we can build whatever intelligence we need into our publishers/consumers. One idea is to have 2 clusters, a primary A and secondary B, have publishers normally publish to A only, and have consumers consume from both A and B. Then to upgrade A, we would need a way to trigger publishers to start publishing to B, and then once all the queues on A were drained by consumers we'd shut it down and upgrade. Then we'd need a similar trigger to switch the publishers back to A.</div>
<div><br></div><div>I think if we define a publishing VIP in our load balancer and have publishers connect to that, we could accomplish the publishing redirection from A to B and back. To switch them to cluster B, we would terminate those VIP connections and point the VIP to cluster B, and then vice-versa after A is upgraded. We'd just need to make sure the publishers automatically reconnect to the VIP after being disconnected.</div>
<div><br></div><div>Sound right?</div><div><br></div><div>thx,</div><div>Chris</div><div class="HOEnZb"><div class="h5"><div><br><div class="gmail_quote">On Wed, Dec 5, 2012 at 7:19 PM, McIntosh Jason <span dir="ltr"><<a href="mailto:mcintoshj@gmail.com" target="_blank">mcintoshj@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Design we use:<br>
<br>
Shovel to a virtual ip rabbit cluster.<br>
Rabbit consumers on that cluster.<br>
Start new cluster.<br>
Can point consumers to hard coded cluster.<br>
Shift shovel and/or change VIP to point to new cluster.<br>
When all messages on old consumers cleaned, shut them down point em to new cluster.<br>
<br>
Our consumers are java apps built into rpms we deploy, so updates are just a restart/new rpm deploy<br>
<br>
This is pretty rough but hope it helps!<br>
Jason<br>
<br>
Sent from my iPhone<br>
<div><div><br>
On Dec 5, 2012, at 9:00 PM, Chris Toomey <<a href="mailto:ctoomey@gmail.com" target="_blank">ctoomey@gmail.com</a>> wrote:<br>
<br>
> Belated follow-up on this thread. Can you talk more about how you manage<br>
> this? E.g., how do you ensure that clients continue consuming from the old<br>
> cluster until all the messages published to it have been consumed? And<br>
> during the transition time, consuming clients consume from both old and new<br>
> cluster, right? What hardware/software do you use to gradually shift<br>
> publishing from old to new cluster?<br>
><br>
> We're just now designing and implementing our infrastructure and apps so<br>
> would like to get it as right as possible from the beginning. Anyone else<br>
> using/recommend other approaches to this problem?<br>
><br>
> thx,<br>
> Chris<br>
><br>
><br>
> Laing, Michael P. wrote<br>
>> We bring up parallel infrastructure with a complete new cluster and<br>
>> gradually shift load to it using weighted routing.<br>
>><br>
>> This won't work for everybody, but we have designed our apps with this in<br>
>> mind.<br>
>><br>
>> Michael<br>
>><br>
>> From: Chris Toomey &lt;<br>
><br>
>> ctoomey@<br>
><br>
>> &lt;mailto:<br>
><br>
>> ctoomey@<br>
><br>
>> &gt;><br>
>> ...<br>
>> That's unfortunate about having to shut down the whole cluster to upgrade<br>
>> it -- it means that our applications will need to have some additional HA<br>
>> queueing mechanism upstream to buffer up the messages to be published<br>
>> during the downtime :-(.<br>
>><br>
>> What kinds of solutions are people using for that problem?<br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> View this message in context: <a href="http://rabbitmq.1065348.n5.nabble.com/High-availability-questions-tp23690p23889.html" target="_blank">http://rabbitmq.1065348.n5.nabble.com/High-availability-questions-tp23690p23889.html</a><br>
> Sent from the RabbitMQ mailing list archive at Nabble.com.<br>
> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Jason McIntosh<br><a href="http://mcintosh.poetshome.com/blog/">http://mcintosh.poetshome.com/blog/</a><br>573-424-7612<br>
</div>