Following up on this, I've gone back and looked at all the logs I can think of.<br><br>On the node that hangs (stuck on "starting database ..."), here's the console output:<br><br>+---+ +---+<br>| | | |<br>
| | | |<br>| | | |<br>| +---+ +-------+<br>| |<br>| RabbitMQ +---+ |<br>| | | |<br>| v2.7.1 +---+ |<br>| |<br>+-------------------+<br>AMQP 0-9-1 / 0-9 / 0-8<br>
Copyright (C) 2007-2011 VMware, Inc.<br>Licensed under the MPL. See <a href="http://www.rabbitmq.com/">http://www.rabbitmq.com/</a><br><br>node : rabbit@play<br>app descriptor : /usr/lib/rabbitmq/lib/rabbitmq_server-2.7.1/sbin/../ebin/rabbit.app<br>
home dir : /home/mpietrek<br>config file(s) : /home/mpietrek/work/var/run/rabbitmq.config<br>cookie hash : pR5H9kY3Wra/XdLELT5hgQ==<br>log : /home/mpietrek/work/logs/<a href="http://play.mpietrek.internal.illumita.com/rabbit@play.log">play.mpietrek.internal.illumita.com/rabbit@play.log</a><br>
sasl log : /home/mpietrek/work/logs/<a href="http://play.mpietrek.internal.illumita.com/rabbit@play-sasl.log">play.mpietrek.internal.illumita.com/rabbit@play-sasl.log</a><br>database dir : /home/mpietrek/work/var/lib/rabbit@play<br>
erlang version : 5.7.4<br><br>-- rabbit boot start<br>starting file handle cache server ...done<br>starting worker pool ...done<br>starting database <br>
<br><br>And this is the last output in the log file:<br><br>=INFO REPORT==== 14-Mar-2012::09:50:33 ===<br>Limiting to approx 924 file handles (829 sockets)<br><br><br>On the node that's the master (labeled "disc stats" in the Overview tab), there's nothing in the log about the new node joining.<br>
<br>Is there anyplace else I should be looking for clues to assist you? This issue is a pretty big spanner in the works for our rolling upgrade scenario.<br><br>Thanks much,<br><br>Matt<br><br><div class="gmail_quote">On Tue, Mar 13, 2012 at 3:00 PM, Matt Pietrek <span dir="ltr"><<a href="mailto:mpietrek@skytap.com">mpietrek@skytap.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Some other work came up so I needed to drop this thread for a few weeks. However, coming back to it, I can easily reproduce this issue within one or two tries.<br>
<br>In a nutshell, in a clustered environment, simply stop one node, wait a few seconds, then restart it. The last output seen is:<br>
<br>starting database <div> ...<br><br></div>I've let it wait for much longer than 30 seconds and it has never come back.<br><br>Any chance this may have been stamped out in RabbitMQ 2.8? <br>
<div class="HOEnZb"><div class="h5">
<br><br><br><div class="gmail_quote">On Fri, Feb 24, 2012 at 1:43 PM, Matt Pietrek <span dir="ltr"><<a href="mailto:mpietrek@skytap.com" target="_blank">mpietrek@skytap.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>| So how long are you waiting when determining it's hanging? Less than 30 seconds?<br><br></div>Just to be double sure, I let it sit for an hour yesterday. I would have expected a timeout, but it never came.<br>
<br>It's a pretty easy scenario to script and try out. I'd send you my code, but it relies on other internal commands.<br>
<br>There may also be a timing issue. If I put a 10 second delay after restarting one broker, and before stopping the next, it seems to help.<br><br>That is:<br><br>for x in broker_list:<br> stop x<br> start x<br> sleep(10)<span><font color="#888888"><br>
<br>Matt</font></span><div><div><br><br><div class="gmail_quote">On Fri, Feb 24, 2012 at 4:22 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 23/02/12 21:00, Matt Pietrek wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The nohup.out on the failing node ends with:<br>
</blockquote>
<br></div>
<snip><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
starting database ...<br>
</blockquote>
<br>
So how long are you waiting when determining it's hanging? Less than 30 seconds?<br>
<br>
Because that looks like Rabbit is waiting for another cluster node (if it was not the last to shut down, but is the first to start up, it will wait for the one that was the last to shut down. But it will only wait for 30 seconds before spitting out an error. I'm not sure how else you could get it to stop there *without* any further output though.<div>
<div><br>
<br>
Cheers, Simon<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>