<html><head><base href="x-msg://218/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>Hi</div><div><br></div><div>On 29 Aug 2012, at 13:12, Pankaj Mishra wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-IN" link="blue" vlink="purple"><div class="Section1" style="page: Section1; "><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Hi,<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">We experienced a strange problem with rabbitmq server running in cluster. Actually according to<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Log file the master of the server crashed. Post that all my publishers continue to send message without<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Throwing any exception but all those messages were dropped silently by rabbitmq server. Consumer were not able<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">To get any of those messages until we restarted the rabbitmq server again.<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">I have attached with this mail the server crash log for master as well as for slave.<o:p></o:p></span></font></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><i><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; font-style: italic; "></span></font></i></div></div></div></span></blockquote><div><br></div><div>According to the master log, the mnesia database has become inconsistent, which is not a good sign. It looks very much like a network partition has occurred here:</div><div><br></div><div>%% from master.log</div><div><div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:24 ===</div><div>rabbit on node rabbit@MyTimes160 down</div></div></div><div><br></div><div>%% from slave.log</div><div><br></div><div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>rabbit on node rabbit@MyTimes159 down</div></div><div><br></div><div>According to the logs, both the master and the slave observed the other node disappear, which seems consistent with the network partition theory.</div><div><br></div><div><div>%% from master.log</div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'cms' in vhost '/'): Master <rabbit@MyTimes159.3.444.0> saw deaths of mirrors <rabbit@MyTimes160.2.595.0> </div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'mytimes' in vhost '/'): Master <rabbit@MyTimes159.3.437.0> saw deaths of mirrors <rabbit@MyTimes160.2.591.0> </div><div><br></div><div>%% from slave.log</div><div><br></div><div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'mytimes' in vhost '/'): Slave <rabbit@MyTimes160.2.591.0> saw deaths of mirrors <rabbit@MyTimes159.3.437.0> </div><div><br></div><div>=INFO REPORT==== 7-Aug-2012::18:42:25 ===</div><div>Mirrored-queue (queue 'cms' in vhost '/'): Slave <rabbit@MyTimes160.2.595.0> saw deaths of mirrors <rabbit@MyTimes159.3.444.0> </div></div><div><br></div><div><br></div><div>Rabbit is not partition tolerant, so I would expect things might go wrong under such circumstances, but I would not expect messages to be silently dropped. My reading of the logs so far is that when the partitioned database state is reached, a message is sent to the gm ring on the 'master' node (the {mnesia_locker,rabbit@MyTimes160,granted} message) which isn't handled, thereby crashing the gm handling process. Once that is down, other things start to go wrong. The parent supervisor will have restarted the failed process to get things back into a consistent state, but it looks as though because mnesia has its knickers in a twist about the partitioned database, that the recovery can't take place properly.</div><div><br></div><div>We will look into this asap, but can you confirm that a net split did in fact take place around the time this problem started appearing?</div><div> </div><div><br></div></div></div></body></html>