<div dir="ltr">Simon,<div><br></div><div>Glad you were able to repro mgmt database thing. </div><div><br></div><div>As for the prior Autoheal activity, here's a sanitized version of what I've got by grepping for "Autoheal", with a little bit of extra context (-C 10). Let me know if you need something else in particular.</div>
<div><br></div><div>On the "winning" side ("goodnessmq1")</div><div><br></div><div>---------------------------------</div><div><div>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:18 ===</p>
<p class="">closing AMQP connection <0.25083.0> (<a href="http://goodness2mq1.foo.bar.com:36360">goodness2mq1.foo.bar.com:36360</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 2-Feb-2014::07:28:18 ===</p>
<p class="">closing AMQP connection <0.19158.0> (<a href="http://goodnessawsd1.foo.bar.com:52133">goodnessawsd1.foo.bar.com:52133</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>):</p>
<p class="">connection_closed_abruptly</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Autoheal request received from rabbit@goodnessmq2</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Mnesia(rabbit@goodnessmq1): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq2}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Autoheal decision</p>
<p class=""> * Partitions: [[rabbit@goodnessmq2],[rabbit@goodnessmq1]]</p>
<p class=""> * Winner: rabbit@goodnessmq1</p>
<p class=""> * Losers: [rabbit@goodnessmq2]</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Autoheal request sent to rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Autoheal: I am the winner, waiting for [rabbit@goodnessmq2] to stop</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Autoheal request received from rabbit@goodnessmq1 when in state {winner_waiting,</p>
<p class=""> [rabbit@goodnessmq2],</p>
<p class=""> [rabbit@goodnessmq2]}; ignoring</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">accepting AMQP connection <0.25103.0> (<a href="http://goodness1mq1.foo.bar.com:57781">goodness1mq1.foo.bar.com:57781</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">connection <0.25103.0>, channel 1 - soft error:</p>
<p class="">{amqp_error,not_found,</p>
<p class=""> "home node 'rabbit@goodnessmq2' of durable queue 'federation: skytap -> mq_prod/goodness1' in vhost '/' is down or inaccessible",</p>
<p class="">--</p>
<p class=""> "home node 'rabbit@goodnessmq2' of durable queue 'federation: skytap -> mq_prod/goodness2' in vhost '/' is down or inaccessible",</p>
<p class=""> 'queue.declare'}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:24 ===</p>
<p class="">closing AMQP connection <0.25176.0> (<a href="http://goodness2mq1.foo.bar.com:59858">goodness2mq1.foo.bar.com:59858</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:25 ===</p>
<p class="">rabbit on node rabbit@goodnessmq2 down</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:25 ===</p>
<p class="">Autoheal: final node has stopped, starting...</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:25 ===</p>
<p class="">rabbit on node rabbit@goodnessmq2 up</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:25 ===</p>
<p class="">accepting AMQP connection <0.25238.0> (<a href="http://goodnessawsd1.foo.bar.com:52215">goodnessawsd1.foo.bar.com:52215</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:26 ===</p>
<p class="">accepting AMQP connection <0.25266.0> (<a href="http://goodness1mq1.foo.bar.com:36928">goodness1mq1.foo.bar.com:36928</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">--</p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">rabbit on node rabbit@goodnessmq2 down</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mnesia(rabbit@goodnessmq1): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq2}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">closing AMQP connection <0.530.4> (<a href="http://goodnessawsd2.foo.bar.com:37902">goodnessawsd2.foo.bar.com:37902</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal request sent to rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal request received from rabbit@goodnessmq2</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">closing AMQP connection <0.777.4> (<a href="http://goodnessawsd2.foo.bar.com:37917">goodnessawsd2.foo.bar.com:37917</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Master <rabbit@goodnessmq1.2.297.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11858.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'accounting' in vhost '/'): Master <rabbit@goodnessmq1.2.292.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11848.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'awsdriver_cron' in vhost '/'): Master <rabbit@goodnessmq1.2.289.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11842.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'mqcmd' in vhost '/'): Master <rabbit@goodnessmq1.2.290.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11844.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal decision</p>
<p class=""> * Partitions: [[rabbit@goodnessmq1],[rabbit@goodnessmq2]]</p>
<p class=""> * Winner: rabbit@goodnessmq1</p>
<p class=""> * Losers: [rabbit@goodnessmq2]</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal request received from rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'system_test' in vhost '/'): Master <rabbit@goodnessmq1.2.298.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11860.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'atropos' in vhost '/'): Master <rabbit@goodnessmq1.2.293.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11850.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'cmcmd' in vhost '/'): Master <rabbit@goodnessmq1.2.296.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11856.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'greenbox' in vhost '/'): Master <rabbit@goodnessmq1.2.295.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11854.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'web-tasks-prod' in vhost '/'): Master <rabbit@goodnessmq1.2.291.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11846.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'quota_inspector' in vhost '/'): Master <rabbit@goodnessmq1.2.294.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11852.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal decision</p>
<p class=""> * Partitions: [[rabbit@goodnessmq1],[rabbit@goodnessmq2]]</p>
<p class=""> * Winner: rabbit@goodnessmq1</p>
<p class=""> * Losers: [rabbit@goodnessmq2]</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal: I am the winner, waiting for [rabbit@goodnessmq2] to stop</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal: I am the winner, waiting additionally for [rabbit@goodnessmq2] to stop</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'awsdriver' in vhost '/'): Master <rabbit@goodnessmq1.2.288.0> saw deaths of mirrors <rabbit@goodnessmq2.1.11840.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">closing AMQP connection <0.998.4> (<a href="http://goodnessawsd2.foo.bar.com:37932">goodnessawsd2.foo.bar.com:37932</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">accepting AMQP connection <0.1025.4> (<a href="http://goodnessawsd2.foo.bar.com:37933">goodnessawsd2.foo.bar.com:37933</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>)</p>
<p class=""><br></p>
<p class="">--</p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 3-Feb-2014::09:48:55 ===</p>
<p class="">closing AMQP connection <0.25414.14> (<a href="http://goodnesswfe2.foo.bar.com:37330">goodnesswfe2.foo.bar.com:37330</a> -> <a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a>):</p>
<p class="">{heartbeat_timeout,running}</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">Mnesia(rabbit@goodnessmq1): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq2}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">Autoheal request received from rabbit@goodnessmq2 when in state {winner_waiting,</p>
<p class=""> [rabbit@goodnessmq2],</p>
<p class=""> [rabbit@goodnessmq2]}; ignoring</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">global: Name conflict terminating {rabbit_mgmt_db,<2783.10073.5>}</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodness1mqvip1.foo.bar.com:5672">goodness1mqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">--</p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::13:14:20 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodness1mqvip1.foo.bar.com:5672">goodness1mqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 3-Feb-2014::13:14:24 ===</p>
<p class="">Mnesia(rabbit@goodnessmq1): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq2}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 3-Feb-2014::13:14:24 ===</p>
<p class="">Autoheal request received from rabbit@goodnessmq2 when in state {winner_waiting,</p>
<p class=""> [rabbit@goodnessmq2],</p>
<p class=""> [rabbit@goodnessmq2]}; ignoring</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::13:14:25 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodness2mqvip1.foo.bar.com:5672">goodness2mqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::13:14:25 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://sgp1r1mqvip1.foo.bar.com:5672">sgp1r1mqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class=""><span class="">17:19</span> <span class="">PROD</span> <span class="">highland@goodnessmq1</span>:~/logs/<a href="http://goodnessmq1.foo.bar.com">goodnessmq1.foo.bar.com</a>$ </p></div><div>---------------------------------</div>
</div><div><br></div><div>And on the losing side ("goodnessmq2"):</div><div><br></div><div><div>---------------------------------</div></div><div>
<p class="">=WARNING REPORT==== 2-Feb-2014::07:28:08 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://sgp1r1mqvip1.foo.bar.com:5672">sgp1r1mqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Mnesia(rabbit@goodnessmq2): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq1}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Autoheal request sent to rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Autoheal: we were selected to restart; winner is rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">Stopping RabbitMQ</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::07:28:20 ===</p>
<p class="">stopped TCP Listener on <a href="http://0.0.0.0:5672">0.0.0.0:5672</a></p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 2-Feb-2014::07:28:21 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://sgp1r1mqvip1.foo.bar.com:5672">sgp1r1mqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class="">--</p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Slave <rabbit@goodnessmq2.1.11858.0> saw deaths of mirrors <rabbit@goodnessmq1.2.297.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Promoting slave <rabbit@goodnessmq2.1.11858.0> to master</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Statistics database started.</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal request sent to rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'awsdriver_cron' in vhost '/'): Slave <rabbit@goodnessmq2.1.11842.0> saw deaths of mirrors <rabbit@goodnessmq1.2.289.0> </p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Mirrored-queue (queue 'awsdriver_cron' in vhost '/'): Promoting slave <rabbit@goodnessmq2.1.11842.0> to master</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Autoheal: we were selected to restart; winner is rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Stopping RabbitMQ</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:06 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' connected to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a></p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 2-Feb-2014::08:55:07 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' connected to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a></p>
<p class=""><br></p>
<p class="">--</p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::09:48:55 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">Mnesia(rabbit@goodnessmq2): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq1}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">Autoheal request sent to rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a></p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p class="">Statistics database started.</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::09:48:58 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a></p>
<p class="">--</p>
<p class="">{error,unknown_host}</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::13:14:24 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a></p>
<p class="">{error,ehostunreach}</p>
<p class=""><br></p>
<p class="">=ERROR REPORT==== 3-Feb-2014::13:14:24 ===</p>
<p class="">Mnesia(rabbit@goodnessmq2): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq1}</p>
<p class=""><br></p>
<p class="">=INFO REPORT==== 3-Feb-2014::13:14:24 ===</p>
<p class="">Autoheal request sent to rabbit@goodnessmq1</p>
<p class=""><br></p>
<p class="">=WARNING REPORT==== 3-Feb-2014::13:14:25 ===</p>
<p class="">Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://goodnessmqvip1.foo.bar.com:5672">goodnessmqvip1.foo.bar.com:5672</a></p>
<p class="">{{shutdown,</p>
<p class=""> {server_initiated_close,404,</p>
<p class=""> <<"NOT_FOUND - no exchange 'federation: skytap -> mq_prod B' in vhost '/'">>}},</p>
<p class=""> {gen_server,call,</p>
<p class=""> [<0.6585.1>,</p>
<p class=""> {call,</p>
<p class=""> {'exchange.bind',0,<<"federation: skytap -> mq_prod B">>,</p></div><div><div>---------------------------------</div></div><div><br></div><div><br></div><div><br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Feb 11, 2014 at 2:15 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On 11/02/14 01:33, Matt Pietrek wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Recently we started running a two node HA cluster of Rabbit 3.2.2, with<br>
autoheal enabled.<br>
<br>
After a network partition, I noticed that autoheal didn't appear to<br>
work, although the logs indicate it was tried. The first time it<br>
happened, the UI in both brokers indicated the other broker was missing<br>
from the cluster.<br>
</blockquote>
<br></div>
So the log indicates that the winning node ignored a request to start autohealing because it was already autohealing.<br>
<br>
It's possible that there is a problem if a network partition occurs while autoheal is already happening. I'll file a bug to look into this, but it would help if you can show me any previous logs from this node - I assume that earlier (probably not much earlier) in the logs there were some more partition warnings and autoheal events?<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The second time this happened, the management plugin seemed to not<br>
function afterwards. Most of the Web UI was unusable, i.e it wouldn't<br>
tell me which nodes were running, what queues were declared, and so forth.<br>
</blockquote>
<br></div>
Separately there is an issue where the management database might fail to recover after a network partition. I just replicated that yesterday; note that it's not connected to autoheal.<br>
<br>
Cheers, Simon<span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</font></span></blockquote></div><br></div>