<div dir="ltr">were you using the federation plugin?<br><br>On Thursday, March 20, 2014 5:41:22 AM UTC-4, Geoffrey Samper wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div dir="ltr"><div><span style="line-height:18px">Hi, </span><br></div><div><p class="MsoNormal"><br></p><p class="MsoNormal">We have a cluster with 3 nodes (Rabbit 1, Rabbit 3, rabbit 4). Sometimes we have problems of network partitioning. Over the day everything runs fine but at night 3AM and 4AM arround we expierence network partitioning. Most of the time between 3 and 1. What strange is the log of rabbit 1 is contains  "Generic server <0.263.0> terminating in the log". <br></p><p class="MsoNormal">Can anyone help me with this issue. The logs are below</p><p class="MsoNormal"><br></p><p class="MsoNormal">RABBIT 04</p><p class="MsoNormal">===========</p><p class="MsoNormal">INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.PageDeletedConsumer' in vhost 'A105'): Master <rabbit@P04.1.264.0> saw deaths of mirrors <rabbit@P01.3.264.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p14' in vhost 'A105'): Master <rabbit@P04.1.262.0> saw deaths of mirrors <rabbit@P01.3.262.0><br><br></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.MNT_grbhstmntp01' in vhost 'A105'): Master <rabbit@P04.1.292.0> saw deaths of mirrors <rabbit@P01.3.292.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p05' in vhost 'A105'): Slave <rabbit@P04.1.24486.0> saw deaths of mirrors <rabbit@P01.3.30101.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p09' in vhost 'A105'): Master <rabbit@P04.1.24669.0> saw deaths of mirrors <rabbit@P01.3.30387.0></p><p class="MsoNormal">Rabbit 03<br>=====================<br>=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>rabbit on node rabbit@P01 down</p><p class="MsoNormal">=ERROR REPORT==== 20-Mar-2014::03:41:27 ===<br>Mnesia(rabbit@P03): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@P01}</p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p21' in vhost 'A105'): Slave <rabbit@P03.2.278.0> saw deaths of mirrors <rabbit@P01.3.275.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p22' in vhost 'A105'): Slave <rabbit@P03.2.270.0> saw deaths of mirrors <rabbit@P01.3.268.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br></p><p class="MsoNormal">Mirrored-queue (queue 'ha.PageDeletedConsumer' in vhost 'A105.001'): Slave <rabbit@P03.2.276.0> saw deaths of mirrors <rabbit@P01.3.274.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p04' in vhost 'A105.001'): Slave <rabbit@P03.2.274.0> saw deaths of mirrors <rabbit@P01.3.271.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.MNT_grbhstmntp02' in vhost 'A105'): Slave <rabbit@P03.2.282.0> saw deaths of mirrors <rabbit@P01.3.280.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p03' in vhost 'A105.001'): Slave <rabbit@P03.2.280.0> saw deaths of mirrors <rabbit@P01.3.278.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.MNT_grbhstmntp01' in vhost 'A105.001'): Slave <rabbit@P03.2.286.0> saw deaths of mirrors <rabbit@P01.3.284.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.services_p20' in vhost 'A105.001'): Slave <rabbit@P03.2.272.0> saw deaths of mirrors <rabbit@P01.3.270.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:27 ===<br>Mirrored-queue (queue 'ha.PageDeleteConsumer' in vhost 'A105.001'): Slave <rabbit@P03.2.284.0> saw deaths of mirrors <rabbit@P01.3.282.0></p><p class="MsoNormal">Rabbit 01<br>==============================<wbr>=====</p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:14 ===<br>rabbit on node rabbit@P03 down</p><p class="MsoNormal">=ERROR REPORT==== 20-Mar-2014::03:41:15 ===<br>Mnesia(rabbit@P01): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@P03}</p><p class="MsoNormal"><br></p><p class="MsoNormal">=ERROR REPORT==== 20-Mar-2014::03:41:15 ===</p><p class="MsoNormal">** Generic server <0.263.0> terminating</p><p class="MsoNormal">** Last message in was {'DOWN',#Ref<0.0.0.1692>,<wbr>process,<4625.265.0>,</p><p class="MsoNormal">                               noconnection}</p><p class="MsoNormal">** When Server state == {state,</p><p class="MsoNormal">                            {31,<0.263.0>},</p><p class="MsoNormal">                            {{28,<4625.265.0>},#Ref<0.0.0.<wbr>1692>},</p><p class="MsoNormal">                            {{25,<5158.263.0>},#Ref<0.0.0.<wbr>1693>},</p><p class="MsoNormal">                            {resource,<<"A105">>,queue,</p><p class="MsoNormal">                                <<"ha.services_p14">>},</p><p class="MsoNormal">                            rabbit_mirror_queue_slave,</p><p class="MsoNormal">                            {32,</p><p class="MsoNormal">                             [{{25,<5158.263.0>},</p><p class="MsoNormal">                               {view_member,</p><p class="MsoNormal">                                   {25,<5158.263.0>},</p><p class="MsoNormal">                                   [],</p><p class="MsoNormal">                                   {31,<0.263.0>},</p><p class="MsoNormal">                                   {28,<4625.265.0>}}},</p><p class="MsoNormal">                              {{28,<4625.265.0>},</p><p class="MsoNormal">                               {view_member,</p><p class="MsoNormal">                                   {28,<4625.265.0>},</p><p class="MsoNormal">                                   [],</p><p class="MsoNormal">                                   {25,<5158.263.0>},</p><p class="MsoNormal">                                   {31,<0.263.0>}}},</p><p class="MsoNormal">                              {{31,<0.263.0>},</p><p class="MsoNormal">                               {view_member,</p><p class="MsoNormal">                                   {31,<0.263.0>},</p><p class="MsoNormal">                                   [],</p><p class="MsoNormal">                                   {28,<4625.265.0>},</p><p class="MsoNormal">                                   {25,<5158.263.0>}}}]},</p><p class="MsoNormal">                            0,</p><p class="MsoNormal">                            [{{25,<5158.263.0>},{member,{[<wbr>],[]},6,6}},</p><p class="MsoNormal">                             {{28,<4625.265.0>},{member,{[<wbr>],[]},1,1}},</p><p class="MsoNormal">                             {{31,<0.263.0>},{member,{[],[<wbr>]},0,0}}],</p><p class="MsoNormal">                            [<0.262.0>],</p><p class="MsoNormal">                            {[],[]},</p><p class="MsoNormal">                            [],0,undefined,</p><p class="MsoNormal">                            #Fun<rabbit_misc.execute_<wbr>mnesia_transaction.1>}</p><p class="MsoNormal">** Reason for termination == </p><p class="MsoNormal">** {function_clause,[{orddict,<wbr>fetch,</p><p class="MsoNormal">                              [{31,<0.263.0>},[]],</p><p class="MsoNormal">                              [{file,"orddict.erl"},{line,<wbr>72}]},</p><p class="MsoNormal">                     {gm,check_neighbours,1,[]},</p><p class="MsoNormal">                     {gm,handle_info,2,[]},</p><p class="MsoNormal">                     {gen_server2,handle_msg,2,[]}<wbr>,</p><p class="MsoNormal">                     {proc_lib,wake_up,3,[{file,"<wbr>proc_lib.erl"},{line,249}]}]}</p><div><span style="line-height:18px">=INFO REPORT==== 20-Mar-2014::03:41:15 ===</span><br></div><p class="MsoNormal">Mirrored-queue (queue 'ha.MNT_Pages_grbhstmntp01' in vhost 'A105'): Slave <rabbit@P01.3.289.0> saw deaths of mirrors <rabbit@P01.3.289.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:15 ===<br>Mirrored-queue (queue 'ha.services_p17' in vhost 'A105'): Slave <rabbit@P01.3.266.0> saw deaths of mirrors <rabbit@P03.2.268.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:15 ===<br>Mirrored-queue (queue 'ha.services_p06' in vhost 'A105'): Slave <rabbit@P01.3.30366.0> saw deaths of mirrors <rabbit@P03.2.21958.0></p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:15 ===<br>Mirrored-queue (queue 'ha.services_p06' in vhost 'A105'): Promoting slave <rabbit@P01.3.30366.0> to master</p><p class="MsoNormal">=INFO REPORT==== 20-Mar-2014::03:41:16 ===<br>Mirrored-queue (queue 'ha.ElasticSearchConsumer' in vhost 'A105.001'): Slave <rabbit@P01.3.286.0> saw deaths of mirrors <rabbit@P03.2.288.0> </p></div></div></blockquote></div>