Francesco,<br><br>Thanks for the quick reply. A couple of replies/questions:<br><br>If I'm understanding what you're saying, we should be starting up our brokers sequentially. However, in my experience this hasn't worked. For instance, we've seen mq1 stall in its startup, waiting for mq3 to start. But mq3 can't start (per the sequential logic) till mq1 finishes starting up. Per advice I received from you previously (below) we've moved to async startup of the brokers:<br>
<br><a href="http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-June/020689.html">http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-June/020689.html</a><br><br><pre>><i> Question 2
</i>><i> ---------------
</i>><i> Related to the above scenario, is there any danger (after an unplanned
</i>><i> shutdown), in simply letting all the nodes start in parallel and
</i>><i> letting Mnesia's waiting sort out the order? It seems to work OK in my
</i>><i> limited testing so far, but I don't know if we're risking data loss.
</i>
It should be fine, but in general it's better to do cluster operations
sequentially and at one site. In this specific case it should be OK.<br><br></pre>As it stands now, we're in a catch 22 - If we do sequential startup, we run the risk of deadlocking if we start the nodes in the wrong order. But if we do async startup, we run into the problem described in this thread.<br>
<br>--------<br>> Uhm. �It looks like mnesia is detecting a deadlock, and I'm not sure why. �What<br>
> happens if you don't kill it? �Does it terminate by itself, eventually?<br><br>I've let it wait for a good long time (30 minutes +) before killing it.<br><br>Thanks much for your help,<br><br>Matt<br><br><div class="gmail_quote">
On Thu, Jul 26, 2012 at 2:40 AM, Francesco Mazzoli <span dir="ltr"><<a href="mailto:francesco@rabbitmq.com" target="_blank">francesco@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Matt,<br>
<br>
At Wed, 25 Jul 2012 11:48:56 -0700,<br>
<div class="im">Matt Pietrek wrote:<br>
> We have a 3 node cluster (mq1, mq2, mq3) running 2.8.4 supporting a small<br>
> number of HA queues. During startup of the cluster, we start all nodes in<br>
> parallel.<br>
<br>
</div>This is not a good idea when dealing with clustering. �RabbitMQ clustering is<br>
basically a thin layer over mnesia clustering, and we need to do some additional<br>
bookkeeping that is prone to race conditions (e.g. storing the online nodes at<br>
shutdown). �We are putting efforts in making this process more reliable on the<br>
rabbit side.<br>
<br>
For this reason you should always execute clustering operations sequentially.<br>
<div class="im"><br>
> Usually everything works fine. However, we've just recently seen one of the<br>
> nodes (mq3) won't start, i.e., the rabbitmqctl wait <pid> doesn't complete.<br>
><br>
> I can log in to the management UI on mq1 and mq2, so they're at least<br>
> minimally running.<br>
><br>
> Luckily, we've turned on verbose Mnesia logging. here's what the failing node<br>
> (mq3) shows in the console spew:<br>
><br>
</div>> [...]<br>
<div class="im">><br>
> The pattern of "Getting table rabbit_durable_exchange (disc_copies) from node<br>
> rabbit@mq1:" cycles between mq1 and mq2 repeatedly until I kill mq3.<br>
<br>
</div>Uhm. �It looks like mnesia is detecting a deadlock, and I'm not sure why. �What<br>
happens if you don't kill it? �Does it terminate by itself, eventually?<br>
<div class="im"><br>
> What other sort of information can I provide or look for when this situation<br>
> repeats?<br>
<br>
</div>Well, the normal rabbit logs would help.<br>
<br>
--<br>
Francesco * Often in error, never in doubt<br>
</blockquote></div><br>