<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hi</div><div><br></div><div>On further investigation, this error condition (verified by the presence of '{mnesia_locker,rabbit@MyTimes160,granted}' in the gm process mailbox) is indeed the result of a netsplit. In RabbitMQ, clustering (and by association, HA/mirror queues) are not partition tolerant, and therefore netsplits *will* cause errors like this to occur. If you cannot rely on the network links between your clustered nodes then you should consider another approach to distribution, such as federation.</div><div><br></div><div>For more details about this, see the distribution guide (<a href="http://www.rabbitmq.com/distributed.html">http://www.rabbitmq.com/distributed.html</a>) and in particular note these comments from the 'Summary' section:</div><div><br></div><div><h2></h2><table style="position: static; z-index: auto; "><tbody><tr><th>Federation / Shovel</th>
<th>Clustering</th>
</tr>
<tr>
<td></td><td><br></td></tr><tr><td>Chooses Availability and Partition Tolerance from
the <a href="http://en.wikipedia.org/wiki/CAP_theorem">CAP
theorem</a>.
</td>
<td>
Chooses Consistency and Availability from the CAP theorem.
</td></tr></tbody></table><div><br></div></div><div>So if you want Consistency (guarantees) and Availability, you should go with clustering and HA, but if you want Availability *and* Partition tolerance, then Clustering/HA is not the right setup for you. Also if this is happening once a month, then I'd suggest looking at what the network admin team is doing around that time, to see if some kit (or software) is being changed, reconfigured and/or taken offline for maintenance during this time period.</div><div><br></div><div>Cheers,</div><div>Tim </div><br><div><div>On 31 Aug 2012, at 10:46, Tim Watson wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><base href="x-msg://218/"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>Hi</div><div><br></div><div>On 29 Aug 2012, at 13:12, Pankaj Mishra wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div lang="EN-IN" link="blue" vlink="purple"><div class="Section1" style="page: Section1; "><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Hi,<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">We experienced a strange problem with rabbitmq server running in cluster. Actually according to<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Log file the master of the server crashed. Post that all my publishers continue to send message without<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Throwing any exception but all those messages were dropped silently by rabbitmq server. Consumer were not able<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">To get any of those messages until we restarted the rabbitmq server again.<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">I have attached with this mail the server crash log for master as well as for slave.<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><i><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; font-style: italic; "></span></font></i></div></div></div></blockquote><div><br></div><div>According to the master log, the mnesia database has become inconsistent, which is not a good sign. It looks very much like a network partition has occurred here:</div><div><br></div><div>%% from master.log</div><div><div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:24 ===</div><div>rabbit on node rabbit@MyTimes160 down</div></div></div><div><br></div><div>%% from slave.log</div><div><br></div><div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>rabbit on node rabbit@MyTimes159 down</div></div><div><br></div><div>According to the logs, both the master and the slave observed the other node disappear, which seems consistent with the network partition theory.</div><div><br></div><div><div>%% from master.log</div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'cms' in vhost '/'): Master <rabbit@MyTimes159.3.444.0> saw deaths of mirrors <rabbit@MyTimes160.2.595.0> </div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'mytimes' in vhost '/'): Master <rabbit@MyTimes159.3.437.0> saw deaths of mirrors <rabbit@MyTimes160.2.591.0> </div><div><br></div><div>%% from slave.log</div><div><br></div><div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'mytimes' in vhost '/'): Slave <rabbit@MyTimes160.2.591.0> saw deaths of mirrors <rabbit@MyTimes159.3.437.0> </div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'cms' in vhost '/'): Slave <rabbit@MyTimes160.2.595.0> saw deaths of mirrors <rabbit@MyTimes159.3.444.0> </div></div><div><br></div><div><br></div><div>Rabbit is not partition tolerant, so I would expect things might go wrong under such circumstances, but I would not expect messages to be silently dropped. My reading of the logs so far is that when the partitioned database state is reached, a message is sent to the gm ring on the 'master' node (the {mnesia_locker,rabbit@MyTimes160,granted} message) which isn't handled, thereby crashing the gm handling process. Once that is down, other things start to go wrong. The parent supervisor will have restarted the failed process to get things back into a consistent state, but it looks as though because mnesia has its knickers in a twist about the partitioned database, that the recovery can't take place properly.</div><div><br></div><div>We will look into this asap, but can you confirm that a net split did in fact take place around the time this problem started appearing?</div><div> </div><div><br></div></div></div></div>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br></blockquote></div><br></body></html>