[rabbitmq-discuss] Network partitioning

Fri May 30 03:31:05 BST 2014

Or, you can enable "pause_minority" feature.

Best Regards 

On Thursday, March 20, 2014 5:41:22 AM UTC-4, Geoffrey Samper wrote:
>
> Hi, 
>
>
> We have a cluster with 3 nodes (Rabbit 1, Rabbit 3, rabbit 4). Sometimes 
> we have problems of network partitioning. Over the day everything runs fine 
> but at night 3AM and 4AM arround we expierence network partitioning. Most 
> of the time between 3 and 1. What strange is the log of rabbit 1 is 
> contains  "Generic server <0.263.0> terminating in the log". 
>
> Can anyone help me with this issue. The logs are below
>
>
> RABBIT 04
>
> ===========
>
> INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.PageDeletedConsumer' in vhost 'A105'): Master 
> <rabbit at P04.1.264.0> saw deaths of mirrors <rabbit at P01.3.264.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p14' in vhost 'A105'): Master 
> <rabbit at P04.1.262.0> saw deaths of mirrors <rabbit at P01.3.262.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.MNT_grbhstmntp01' in vhost 'A105'): Master 
> <rabbit at P04.1.292.0> saw deaths of mirrors <rabbit at P01.3.292.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p05' in vhost 'A105'): Slave 
> <rabbit at P04.1.24486.0> saw deaths of mirrors <rabbit at P01.3.30101.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p09' in vhost 'A105'): Master 
> <rabbit at P04.1.24669.0> saw deaths of mirrors <rabbit at P01.3.30387.0>
>
> Rabbit 03
> =====================
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> rabbit on node rabbit at P01 down
>
> =ERROR REPORT==== 20-Mar-2014::03:41:27 ===
> Mnesia(rabbit at P03): ** ERROR ** mnesia_event got {inconsistent_database, 
> running_partitioned_network, rabbit at P01}
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p21' in vhost 'A105'): Slave 
> <rabbit at P03.2.278.0> saw deaths of mirrors <rabbit at P01.3.275.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p22' in vhost 'A105'): Slave 
> <rabbit at P03.2.270.0> saw deaths of mirrors <rabbit at P01.3.268.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
>
> Mirrored-queue (queue 'ha.PageDeletedConsumer' in vhost 'A105.001'): Slave 
> <rabbit at P03.2.276.0> saw deaths of mirrors <rabbit at P01.3.274.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p04' in vhost 'A105.001'): Slave 
> <rabbit at P03.2.274.0> saw deaths of mirrors <rabbit at P01.3.271.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.MNT_grbhstmntp02' in vhost 'A105'): Slave 
> <rabbit at P03.2.282.0> saw deaths of mirrors <rabbit at P01.3.280.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p03' in vhost 'A105.001'): Slave 
> <rabbit at P03.2.280.0> saw deaths of mirrors <rabbit at P01.3.278.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.MNT_grbhstmntp01' in vhost 'A105.001'): Slave 
> <rabbit at P03.2.286.0> saw deaths of mirrors <rabbit at P01.3.284.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.services_p20' in vhost 'A105.001'): Slave 
> <rabbit at P03.2.272.0> saw deaths of mirrors <rabbit at P01.3.270.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:27 ===
> Mirrored-queue (queue 'ha.PageDeleteConsumer' in vhost 'A105.001'): Slave 
> <rabbit at P03.2.284.0> saw deaths of mirrors <rabbit at P01.3.282.0>
>
> Rabbit 01
> ===================================
>
> =INFO REPORT==== 20-Mar-2014::03:41:14 ===
> rabbit on node rabbit at P03 down
>
> =ERROR REPORT==== 20-Mar-2014::03:41:15 ===
> Mnesia(rabbit at P01): ** ERROR ** mnesia_event got {inconsistent_database, 
> running_partitioned_network, rabbit at P03}
>
>
> =ERROR REPORT==== 20-Mar-2014::03:41:15 ===
>
> ** Generic server <0.263.0> terminating
>
> ** Last message in was {'DOWN',#Ref<0.0.0.1692>,process,<4625.265.0>,
>
>                                noconnection}
>
> ** When Server state == {state,
>
>                             {31,<0.263.0>},
>
>                             {{28,<4625.265.0>},#Ref<0.0.0.1692>},
>
>                             {{25,<5158.263.0>},#Ref<0.0.0.1693>},
>
>                             {resource,<<"A105">>,queue,
>
>                                 <<"ha.services_p14">>},
>
>                             rabbit_mirror_queue_slave,
>
>                             {32,
>
>                              [{{25,<5158.263.0>},
>
>                                {view_member,
>
>                                    {25,<5158.263.0>},
>
>                                    [],
>
>                                    {31,<0.263.0>},
>
>                                    {28,<4625.265.0>}}},
>
>                               {{28,<4625.265.0>},
>
>                                {view_member,
>
>                                    {28,<4625.265.0>},
>
>                                    [],
>
>                                    {25,<5158.263.0>},
>
>                                    {31,<0.263.0>}}},
>
>                               {{31,<0.263.0>},
>
>                                {view_member,
>
>                                    {31,<0.263.0>},
>
>                                    [],
>
>                                    {28,<4625.265.0>},
>
>                                    {25,<5158.263.0>}}}]},
>
>                             0,
>
>                             [{{25,<5158.263.0>},{member,{[],[]},6,6}},
>
>                              {{28,<4625.265.0>},{member,{[],[]},1,1}},
>
>                              {{31,<0.263.0>},{member,{[],[]},0,0}}],
>
>                             [<0.262.0>],
>
>                             {[],[]},
>
>                             [],0,undefined,
>
>                             #Fun<rabbit_misc.execute_mnesia_transaction.1>}
>
> ** Reason for termination == 
>
> ** {function_clause,[{orddict,fetch,
>
>                               [{31,<0.263.0>},[]],
>
>                               [{file,"orddict.erl"},{line,72}]},
>
>                      {gm,check_neighbours,1,[]},
>
>                      {gm,handle_info,2,[]},
>
>                      {gen_server2,handle_msg,2,[]},
>
>                     
>  {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
> =INFO REPORT==== 20-Mar-2014::03:41:15 ===
>
> Mirrored-queue (queue 'ha.MNT_Pages_grbhstmntp01' in vhost 'A105'): Slave 
> <rabbit at P01.3.289.0> saw deaths of mirrors <rabbit at P01.3.289.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:15 ===
> Mirrored-queue (queue 'ha.services_p17' in vhost 'A105'): Slave 
> <rabbit at P01.3.266.0> saw deaths of mirrors <rabbit at P03.2.268.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:15 ===
> Mirrored-queue (queue 'ha.services_p06' in vhost 'A105'): Slave 
> <rabbit at P01.3.30366.0> saw deaths of mirrors <rabbit at P03.2.21958.0>
>
> =INFO REPORT==== 20-Mar-2014::03:41:15 ===
> Mirrored-queue (queue 'ha.services_p06' in vhost 'A105'): Promoting slave 
> <rabbit at P01.3.30366.0> to master
>
> =INFO REPORT==== 20-Mar-2014::03:41:16 ===
> Mirrored-queue (queue 'ha.ElasticSearchConsumer' in vhost 'A105.001'): 
> Slave <rabbit at P01.3.286.0> saw deaths of mirrors <rabbit at P03.2.288.0> 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140529/736a9ed5/attachment.html>