<div dir="ltr"><div class="gmail_extra"><br><br><div class="gmail_quote">On 11 February 2014 13:45, Ron <span dir="ltr"><<a href="mailto:ron.cordell@gmail.com" target="_blank">ron.cordell@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>We have seen the same behavior but don't have a fix for it. In a 3 node HA cluster we sometimes see node 1 out as seen by node 2, node 2 out as seen by node 1, and node 3 thinks everything is ok. Pivotal Labs was working with us at one point and they didn't have an explanation, either. </div>
<div><br></div><div>That being said we have had numerous issues getting a stable and reliable 3 node cluster working on Windows Server 2008R2. We don't see the stability issues in our tests with Linux but we won't be running production on Linux rabbit nodes for a couple more weeks. </div>
<div><br></div><div>Cheers,</div><div><br></div><div>Ron<br><br>Sent from my iPad</div><div><div class="h5"><div><br>On Feb 10, 2014, at 5:33 PM, Matt Pietrek <<a href="mailto:mpietrek@skytap.com" target="_blank">mpietrek@skytap.com</a>> wrote:<br>
<br></div><blockquote type="cite"><div><div dir="ltr">Recently we started running a two node HA cluster of Rabbit 3.2.2, with autoheal enabled.<div><br></div><div>After a network partition, I noticed that autoheal didn't appear to work, although the logs indicate it was tried. The first time it happened, the UI in both brokers indicated the other broker was missing from the cluster.</div>
<div><br></div><div>The second time this happened, the management plugin seemed to not function afterwards. Most of the Web UI was unusable, i.e it wouldn't tell me which nodes were running, what queues were declared, and so forth.</div>
</div></div></blockquote></div></div></div></blockquote><div><div class="gmail_default" style="font-family:courier new,monospace;display:inline">One thing i learned from the one of the similar discussion in this mailing list you can start up the web ui using<br>
sudo rabbitmqctl eval 'application:start(rabbitmq_management).'<br></div><div class="gmail_default" style="font-family:courier new,monospace;display:inline">
or<br>sudo rabbitmqctl eval 'application:stop(rabbitmq_management).'<br><br></div><div class="gmail_default" style="font-family:courier new,monospace;display:inline">At least that will give you management UI back<br>
</div><div class="gmail_default" style="font-family:courier new,monospace;display:inline">
</div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div><div class="h5"><blockquote type="cite"><div><div dir="ltr">
<div><br></div><div><br></div><div>I'm wondering if what I'm seeing below is a known issue rings any bells. Also, is their any other log output I should look at to determine success/failure?</div><div><br></div><div>
On the "winning" side, the logs look like this. The "ignoring" part in particular is suspicious.</div><div><br></div><div>--------</div><div>
<p>=ERROR REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p>Mnesia(rabbit@goodnessmq1): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq2}</p>
<p><br></p>
<p>=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p>Autoheal request received from rabbit@goodnessmq2 when in state {winner_waiting,</p>
<p> [rabbit@goodnessmq2],</p>
<p> [rabbit@goodnessmq2]}; ignoring</p>
<p><br></p>
<p>=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p>
<p>global: Name conflict terminating {rabbit_mgmt_db,<2783.10073.5>}</p><p>--------</p><p><br></p><p>On the "losing" side, the logs look like this:</p><p>--------</p>
<p>=ERROR REPORT==== 3-Feb-2014::09:48:56 ===</p><p>Mnesia(rabbit@goodnessmq2): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@goodnessmq1}</p><p><br></p>
<p>=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p><p>Autoheal request sent to rabbit@goodnessmq1</p><p><br></p><p>=WARNING REPORT==== 3-Feb-2014::09:48:56 ===</p><p>Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://something <a href="http://else.foo.bar.com:5672" target="_blank">else.foo.bar.com:5672</a></p>
<p>{error,unknown_host}</p><p>=INFO REPORT==== 3-Feb-2014::09:48:56 ===</p><p>Statistics database started.</p><p><br></p><p>=WARNING REPORT==== 3-Feb-2014::09:48:58 ===</p><p>
Federation exchange 'skytap' in vhost '/' did not connect to exchange 'skytap' in vhost '/' on amqp://<a href="http://somethingelse.foo.bar.com:5672" target="_blank">somethingelse.foo.bar.com:5672</a></p>
<p>
</p><p>{error,unknown_host}</p><p>--------</p></div></div></div></blockquote></div></div></div></blockquote><div><br><div class="gmail_default" style="font-family:courier new,monospace;display:inline">Not quite sure what's going on there since we are not using federation, but have you checked if the "losing side" can connect to the <a href="http://somethingelse.foo.bar.com">somethingelse.foo.bar.com</a> at that port ? I remembered when I was playing around with federation and clustering I had issue because one of the node couldn't resolve the other host because it didn't exist in the /etc/hosts or due to firewall issue</div>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div><div class="h5"><blockquote type="cite"><div>
</div></blockquote></div></div><blockquote type="cite"><div><span>_______________________________________________</span><br><span>rabbitmq-discuss mailing list</span><br><span><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a></span><br>
<span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a></span><br></div></blockquote></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div></div>