Alright, thanks, I'll try to contact you on IRC next time this happens. I updated the gist with some logs from another node, the third node produced *a lot*, so those I have to email you.
<br/><br/><br/>On Thursday 6 June 2013 at 18:07, Tim Watson-5 [via RabbitMQ] wrote:
<br/><div class='shrinkable-quote'><br/>> Carl
<br/>>
<br/>> On 5 Jun 2013, at 13:56, carlhoerberg <[hidden email] (/user/SendEmail.jtp?type=node&node=27221&i=0)> wrote:
<br/>> > On a three node cluster, one ec2 machine reboots unexpectedly, and when it
<br/>> > starts up again RabbitMQ fails to start. I've put all logs here:
<br/>> > <a href="https://gist.github.com/carlhoerberg/ff6c6bd4f7639bf4b2f5" target="_top" rel="nofollow" link="external">https://gist.github.com/carlhoerberg/ff6c6bd4f7639bf4b2f5</a><br/>> >
<br/>>
<br/>>
<br/>> That seems to contain only the logs from one node, what about the others?
<br/>>
<br/>> > When the troubled node is restarted manually again it's unable to join,
<br/>> > stopping at "adding mirrors", staying there forever.
<br/>> >
<br/>> > The other nodes now start to behave weird too, new queues can't be declared,
<br/>> > but existing queues seems to continue deliver messages. They also can't
<br/>> > respond to "rabbitmqctl status", or /api/overview. I'm forced to stop them
<br/>> > with "kill -9". Only when all nodes are stopped the cluster can be brought
<br/>> > up again normally.
<br/>> >
<br/>>
<br/>>
<br/>> If you kill -9 the nodes, it's a bit tricky to get live info for diagnosis, assuming there's nothing in the logs. If the logs are available, please post them. Next time this happens, jump on irc (the #rabbitmq channel on freenode) and we can try a few things to diagnose what's going on. If you can arrange for me to have ssh access to these nodes whilst the symptoms are present, I'll be more likely to solve the issue quickly - we might be able to sign some kind of privacy agreement if necessary.
<br/>>
<br/>> Also please post your full setup whenever possible, detailing which plugins you're using (if any) and what kind of ha setup you're using.
<br/>>
<br/>> Cheers,
<br/>> Tim
<br/>> _______________________________________________
<br/>> rabbitmq-discuss mailing list
<br/>> [hidden email] (/user/SendEmail.jtp?type=node&node=27221&i=1)
<br/>> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_top" rel="nofollow" link="external">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br/>>
<br/>>
<br/>> If you reply to this email, your message will be added to the discussion below: <a href="http://rabbitmq.1065348.n5.nabble.com/Node-crash-then-cluster-collapse-tp27206p27221.html" target="_top" rel="nofollow" link="external">http://rabbitmq.1065348.n5.nabble.com/Node-crash-then-cluster-collapse-tp27206p27221.html</a>
<br/>> To unsubscribe from Node crash, then cluster collapse, click here (<a href="" target="_top" rel="nofollow" link="external">
<br/>> NAML (<a href="http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml" target="_top" rel="nofollow" link="external">http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml</a>)
<br/>>
</div><br/><br/><br/>
<br/><hr align="left" width="300" />
View this message in context: <a href="http://rabbitmq.1065348.n5.nabble.com/Node-crash-then-cluster-collapse-tp27206p27239.html">Re: Node crash, then cluster collapse</a><br/>
Sent from the <a href="http://rabbitmq.1065348.n5.nabble.com/">RabbitMQ mailing list archive</a> at Nabble.com.<br/>