<br><br><div class="gmail_quote">On Fri, Feb 4, 2011 at 5:10 AM, Matthew Sackman <span dir="ltr"><<a href="mailto:matthew@rabbitmq.com">matthew@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Mon, Jan 31, 2011 at 07:26:34AM -0800, Bill Moseley wrote:<br>
> For those of you running multiple RabbitMQ servers in a cluster, what is<br>
> your procedure when you want to shut one of the servers down (e.g. for<br>
> maintenance) but not disrupt overall service? Queues only live on one<br>
> server so I'm wondering how (or if) you do something to flush out the queue<br>
> before stopping the machine.<br>
<br>
</div>Usual best practise is to force clients to reconnect elsewhere,<br>
recreating the resources they need. This may need some careful thought<br>
with ordering of events etc. Frequent best practise is that publishers<br>
create exchanges, and consumers create the queues they need and bind<br>
them as necessary. To avoid missing any messages you'll need to start up<br>
new consumers before taking down the old ones.</blockquote><div><br></div><div>Thanks for responding -- I had just this morning thought about this question and here was your response! </div><div><br></div><div>Configuration on the clients is something I'm trying to reduce -- as well as the need to trigger a reconnect on all clients manually. Maybe what I need is better configuration management.</div>
<div><br></div><div>One approach I was considering is having all clients (producers and consumers) connect to a load balancer in front of two independent RabbitMQ brokers. They would not be in a cluster (although each one could be its own cluster of machines for scaling). The balancer only uses one RabbitMQ broker and the second is hot-standby. The consumers connect to a different IP than the producers. Clients know to reconnect upon loss of a connection.</div>
<div><br></div><div>Then the trick is to use the load balancer to make the producers move to the new broker, move some consumers as well, and leave some other consumers to drain the queues on the old broker before pulling out of service.</div>
<div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> But they must create new<br>
queues, not on the to-die node. So this requires the queue names must be<br>
fresh, but then you're going to have to deal with the possibilities of<br>
duplicate messages during the period that multiple sets of consumers are<br>
up etc.<br></blockquote><div><br></div><div>Renaming the queues is only needed if pulling a machine out of cluster (for the queues that were created on that machine), correct? I would not need that with two separate brokers as I describe above, if I follow what you are saying.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">The "or-else" routing semantics of RabbitMQ's "Alternate Exchanges" may<br>
well be of use here.<br>
<a href="http://www.rabbitmq.com/extensions.html#alternate-exchange" target="_blank">http://www.rabbitmq.com/extensions.html#alternate-exchange</a></blockquote><div><br></div><div>Yes, I've been looking at those. Are you saying that if the queue is not durable then once all the consumers of that queue go away then could use the alternate-exchange as a type of fail-over?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">> Now, this is a bit tougher: How about catastrophic failures? I'm wondering<br>
> about using the complexity of Pacemaker and DRBD vs. tracking incomplete<br>
> jobs and resubmitting after some time.<br>
<br>
</div>Horses for courses really. We know of a number of clients who are using<br>
the pacemaker stuff, though frequently with NAS/SAN rather than DRBD. If<br>
you can work out what failures you can withstand and what you can't and<br>
then pick the best approach to match.<br></blockquote><div><br></div><div>The story from our IT department is we don't like DRBD and NAS/SAN is too expensive. ;) But, they want to be able to yank the plug on a RabbitMQ box and have the application continue with no disruption. Basically, no message is ever lost.</div>
<div><br></div><div>Yes, the key might be knowing what failures we can withstand. My current thinking has shifted a bit. I think that instead of trying to build something that never fails, just assume it's not very likely to fail but design the application to handle a queue failure. What that means is that for messages (really "jobs" in this case) that must complete, the producer, or some agent of the producer (perhaps a cron job) will see the job was not completed in a timely way and send a new message.</div>
<div><br></div><div>Maybe that's not so pretty if the queue has a million messages pending when it fails, but if that's the case then there's other more serious problems....</div><div><br></div></div><br>-- <br>
Bill Moseley<br><a href="mailto:moseley@hank.org" target="_blank">moseley@hank.org</a><br>