[rabbitmq-discuss] High availability questions

Jason McIntosh mcintoshj at gmail.com
Thu Dec 6 20:17:33 GMT 2012


So our "publishers" publish to a local rabbitmq in a remote data center.
 Then each shovels their messages to a warehousing cluster for consumption.
 In our case, upgrading a remote system is just pulling that remote system
out of the load balancer, waiting for all it's messages to shovel,
stopping, upgrading it, then rinse & repeat.

In theory, you could do something along the lines of what you're talking
about by clustering your publisher rabbit boxes together, just like you'd
do for the consumer process.  As long as your load balancer stays up
reliably (which they're supposed to do).  I HAVE had issues with connection
resets breaking clients, so like anything else, make sure you test it.
 Connection resets though when shoveling seem to handle such a situation
fine - I'm guessing shovel is designed to work even when it's connection
breaks :)  Which is AWESOME btw.

Jason


On Thu, Dec 6, 2012 at 1:06 PM, Chris Toomey <ctoomey at gmail.com> wrote:

> So your publishers connect to cluster A, which shovels a VIP that
> initially points to cluster B, and your consumers connect to B.  To upgrade
> B, you point the shovel VIP to new cluster C and start new consumers on C,
> then kill off the consumers on B when they've consumed all the messages.
>  Right?  How do you upgrade cluster A?
>
> We'd like to have a solution that doesn't require our publishers/consumers
> to be touched/restarted at all; since we're starting with a clean sheet we
> can build whatever intelligence we need into our publishers/consumers.  One
> idea is to have 2 clusters, a primary A and secondary B, have publishers
> normally publish to A only, and have consumers consume from both A and B.
>  Then to upgrade A, we would need a way to trigger publishers to start
> publishing to B, and then once all the queues on A were drained by
> consumers we'd shut it down and upgrade.  Then we'd need a similar trigger
> to switch the publishers back to A.
>
> I think if we define a publishing VIP in our load balancer and have
> publishers connect to that, we could accomplish the publishing redirection
> from A to B and back.  To switch them to cluster B, we would terminate
> those VIP connections and point the VIP to cluster B, and then vice-versa
> after A is upgraded.  We'd just need to make sure the publishers
> automatically reconnect to the VIP after being disconnected.
>
> Sound right?
>
> thx,
> Chris
>
> On Wed, Dec 5, 2012 at 7:19 PM, McIntosh Jason <mcintoshj at gmail.com>wrote:
>
>> Design we use:
>>
>> Shovel to a virtual ip rabbit cluster.
>> Rabbit consumers on that cluster.
>> Start new cluster.
>> Can point consumers to hard coded cluster.
>> Shift shovel and/or change VIP to point to new cluster.
>> When all messages on old consumers cleaned, shut them down point em to
>> new cluster.
>>
>> Our consumers are java apps built into rpms we deploy, so updates are
>> just a restart/new rpm deploy
>>
>> This is pretty rough but hope it helps!
>> Jason
>>
>> Sent from my iPhone
>>
>> On Dec 5, 2012, at 9:00 PM, Chris Toomey <ctoomey at gmail.com> wrote:
>>
>> > Belated follow-up on this thread.  Can you talk more about how you
>> manage
>> > this?  E.g., how do you ensure that clients continue consuming from the
>> old
>> > cluster until all the messages published to it have been consumed?  And
>> > during the transition time, consuming clients consume from both old and
>> new
>> > cluster, right?  What hardware/software do you use to gradually shift
>> > publishing from old to new cluster?
>> >
>> > We're just now designing and implementing our infrastructure and apps so
>> > would like to get it as right as possible from the beginning.  Anyone
>> else
>> > using/recommend other approaches to this problem?
>> >
>> > thx,
>> > Chris
>> >
>> >
>> > Laing, Michael P. wrote
>> >> We bring up parallel infrastructure with a complete new cluster and
>> >> gradually shift load to it using weighted routing.
>> >>
>> >> This won't work for everybody, but we have designed our apps with this
>> in
>> >> mind.
>> >>
>> >> Michael
>> >>
>> >> From: Chris Toomey &lt;
>> >
>> >> ctoomey@
>> >
>> >> &lt;mailto:
>> >
>> >> ctoomey@
>> >
>> >> &gt;>
>> >> ...
>> >> That's unfortunate about having to shut down the whole cluster to
>> upgrade
>> >> it -- it means that our applications will need to have some additional
>> HA
>> >> queueing mechanism upstream to buffer up the messages to be published
>> >> during the downtime :-(.
>> >>
>> >> What kinds of solutions are people using for that problem?
>> >
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://rabbitmq.1065348.n5.nabble.com/High-availability-questions-tp23690p23889.html
>> > Sent from the RabbitMQ mailing list archive at Nabble.com.
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>


-- 
Jason McIntosh
http://mcintosh.poetshome.com/blog/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121206/f1e29201/attachment.htm>


More information about the rabbitmq-discuss mailing list