Francesco,<br><br>Thanks again for the valuable insight from your reply. I'm down to one issue at this point.<br><br>Given what you said earlier about it being OK to start the brokers in any order, I wrote a simple "catastrophic stress" test. The good news is that RabbitMQ does what's expected. The bad news: Only most of the time, i.e. about 90%.<br>
<br>Here's what the test does:<br><br>In a loop (5 times):<br> Write 5 messages to a queue comprised of 3 clustered nodes (disk based).<br> Kill all RabbitMQ's as quickly as possible - I use "killall beam.smp" invoked on all nodes simultaneously via Capistrano.<br>
Restart all nodes in parallel (again, using Capistrano).<br> Verify that each previously written message can be received.<br><br>~90% of the time, an iteration runs to completion as expected. About 10% of the time, one of the nodes fails to start,<br>
getting stuck with the last line being "starting database ..."<br><br>I see nothing of note in the rabbit@<node>.log file.<br><br>When this happens, I have no luck getting the node to join the cluster. Even attempting a "rabbitmqctl force_reset" hangs. The only way I can get the cluster fully formed again is to terminate all the nodes, then bring them back up. (A reset is not required in this case, however.)<br>
<br>For what it's worth, the broker start logic looks like this<br> nohup rabbitmq-server &<br> rabbitmqctl wait <pidfile><br><br>Are their any trace options or other places to look to understand why a node gets stuck on "creating database..." ?<br>
<br>This bit of startup unpredictability is causing a lot of concern with our VP. I've been tasked with coming up with the complete story, and workarounds to any problems like this.<br><br>Thanks very much again,<br><br>
Matt<br><br><div class="gmail_quote">On Tue, Jun 19, 2012 at 1:49 AM, Francesco Mazzoli <span dir="ltr"><<a href="mailto:francesco@rabbitmq.com" target="_blank">francesco@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Matt,<br>
<br>
At Mon, 18 Jun 2012 11:16:40 -0700,<br>
<div class="im">Matt Pietrek wrote:<br>
><br>
> Francesco,<br>
><br>
> Thanks very much for the detailed reply. It was extremely helpful.<br>
<br>
</div>Glad I could help.<br>
<br>
> Question 1<br>
> ----------------<br>
<div class="im">> I stumbled across this in my own previous experimentation. The<br>
> question is, do I risk message loss by first starting a node that<br>
> joined the cluster late, thus not having the full set of messages that<br>
> other nodes have?<br>
<br>
</div>No.<br>
<div class="im"><br>
> Question 2<br>
> ---------------<br>
> Related to the above scenario, is there any danger (after an unplanned<br>
> shutdown), in simply letting all the nodes start in parallel and<br>
> letting Mnesia's waiting sort out the order? It seems to work OK in my<br>
> limited testing so far, but I don't know if we're risking data loss.<br>
<br>
</div>It should be fine, but in general it's better to do cluster operations<br>
sequentially and at one site. In this specific case it should be OK.<br>
<div class="im"><br>
> Question 3<br>
> ---------------<br>
> You said:<br>
><br>
> > In other words, it's up to you to restart them so that the node with the most up-to-date mnesia is started first<br>
><br>
> Is there any information recorded somewhere (e.g. the logs), which<br>
> would indicate which node has the "most up to date" Mnesia database? I<br>
> see messages like:<br>
><br>
> > Mirrored-queue (queue 'charon' in vhost '/'): Promoting slave<br>
> <rabbit@play2.1.273.0> to master<br>
><br>
> But don't know if they're necessarily correlated to who the most<br>
> up-to-date Mnesia is.<br>
<br>
</div>Well, the fact that something got promoted to master tells you that<br>
the previous master definitely went down before the promoted node.<br>
<br>
There won't be precise information if you stop the nodes abruptly, but<br>
you can look at "rabbit on node <node> down" messages in the logs -<br>
the node in the clusters are linked in distributed Erlang and these<br>
messages are generated when a node receives a 'DOWN' message from<br>
another node. Since they are "info" messages, you need to make sure to<br>
have those enabled - they are enabled by default but a lot of people<br>
don't like them.<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
Francesco.<br>
</font></span></blockquote></div><br>