<div dir="ltr">Hi, Michael:<div><br></div><div>This isn't terribly surprising. Snapshotting a VM is likely to render it less responsive than it normally would be for some amount of time. Whether that period ends before some other node in your cluster misses heartbeats and gives you grief about it, is a coin flip.</div>
<div><br></div><div>Best regards,</div><div>Jerry</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 20, 2014 at 3:12 AM, Michael Oullion <span dir="ltr"><<a href="mailto:michael.oullion@norbert-dentressangle.com" target="_blank">michael.oullion@norbert-dentressangle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all, <div><br></div><div>We observe some net split on our cluster and we don't know why.</div><div>
Before change the net tick parameter and change the net split behavior, I want to understand why it's happening.</div>
<div>Our environment is : </div><div>RabbitMQ 3.2.1 Elrang R16B</div><div>3 RabbitMQ Node in the same sub-network</div><div>RabbitMQ is installed on Windows 2008 R2 (VMWare ESXi 5.1)</div><div>We have 4 Mirrored Queues on this cluster.</div>
<div>In production, the normal stream is about 20 messages/second.</div><div><br></div><div>We observe that split occurs always at the end of the snapshot (NetBackup) on the VM.</div><div>But, we made snapshot each night and the network split occurs 1 time each 15 or 20 days.</div>
<div><br></div><div><b><u>Log server rabbit@FRA-VSP-32545 :</u></b></div><div><div>=INFO REPORT==== 19-Feb-2014::18:34:47 ===</div><div>rabbit on node 'rabbit@FRA-VSP-32596' down</div><div><br></div><div>=INFO REPORT==== 19-Feb-2014::18:34:49 ===</div>
<div>Mirrored-queue (queue 'conso.queue.dead' in vhost '/IEC'): Slave <'rabbit@FRA-VSP-32545'.2.269.0> saw deaths of mirrors <'rabbit@FRA-VSP-32596'.1.270.0> </div></div><div><br>
</div><div><br></div><div><b><u>Log server rabbit@FRA-VSP-32596 :</u></b></div><div><div><br></div><div>=INFO REPORT==== 19-Feb-2014::18:34:28 ===</div><div>rabbit on node 'rabbit@FRA-VSP-32545' down</div><div><br>
</div><div>=ERROR REPORT==== 19-Feb-2014::18:34:30 ===</div><div>** Generic server <0.279.0> terminating</div><div>** Last message in was {'DOWN',#Ref<0.0.0.248452>,process,<5383.278.0>,</div><div>
noconnection}</div><div>** When Server state == {state,</div><div> {76,<0.279.0>},</div><div> {{79,<5383.278.0>},#Ref<0.0.0.248452>},</div>
<div> {{82,<5066.278.0>},#Ref<0.0.1.42330>},</div><div> {resource,<<"/IEC">>,queue,<<"conso.queue">>},</div><div>
rabbit_mirror_queue_coordinator,</div><div> {83,</div><div> [{{76,<0.279.0>},</div><div> {view_member,</div>
<div> {76,<0.279.0>},</div><div> [],</div><div> {79,<5383.278.0>},</div><div> {82,<5066.278.0>}}},</div>
<div> {{79,<5383.278.0>},</div><div> {view_member,</div><div> {79,<5383.278.0>},</div><div> [],</div>
<div> {82,<5066.278.0>},</div><div> {76,<0.279.0>}}},</div><div> {{82,<5066.278.0>},</div><div> {view_member,</div>
<div> {82,<5066.278.0>},</div><div> [],</div><div> {76,<0.279.0>},</div><div> {79,<5383.278.0>}}}]},</div>
<div> 1457518,</div><div> [{{76,<0.279.0>},{member,{[],[]},1457518,1457518}},</div><div> {{79,<5383.278.0>},{member,{[],[]},1,1}},</div>
<div> {{82,<5066.278.0>},{member,{[],[]},0,0}}],</div><div> [<0.1272.0>],</div><div> {[],[]},</div><div> [],undefined,</div>
<div> #Fun<rabbit_misc.execute_mnesia_transaction.1>}</div><div>** Reason for termination == </div><div>** {function_clause,</div><div> [{orddict,fetch,</div><div> [{76,<0.279.0>},</div>
<div> [{{82,<5066.278.0>},</div><div> {view_member,</div><div> {82,<5066.278.0>},</div><div> [{79,<5383.278.0>}],</div><div> {82,<5066.278.0>},</div>
<div> {82,<5066.278.0>}}}]],</div><div> [{file,"orddict.erl"},{line,72}]},</div><div> {gm,check_neighbours,1,[]},</div><div> {gm,handle_info,2,[]},</div><div> {gen_server2,handle_msg,2,[]},</div>
<div> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</div><div><br></div><div>=ERROR REPORT==== 19-Feb-2014::18:34:30 ===</div><div>** Generic server <0.283.0> terminating</div><div>** Last message in was {'DOWN',#Ref<0.0.0.248454>,process,<5383.282.0>,</div>
<div> noconnection}</div><div>** When Server state == {state,</div><div> {67,<0.283.0>},</div><div> {{70,<5383.282.0>},#Ref<0.0.0.248454>},</div>
<div> {{73,<5066.282.0>},#Ref<0.0.1.42352>},</div><div> {resource,<<"/IEC">>,queue,<<"event.queue">>},</div><div>
rabbit_mirror_queue_coordinator,</div><div> {74,</div><div> [{{67,<0.283.0>},</div><div> {view_member,</div>
<div> {67,<0.283.0>},</div><div> [],</div><div> {70,<5383.282.0>},</div><div> {73,<5066.282.0>}}},</div>
<div> {{70,<5383.282.0>},</div><div> {view_member,</div><div> {70,<5383.282.0>},</div><div> [],</div>
<div> {73,<5066.282.0>},</div><div> {67,<0.283.0>}}},</div><div> {{73,<5066.282.0>},</div><div> {view_member,</div>
<div> {73,<5066.282.0>},</div><div> [],</div><div> {67,<0.283.0>},</div><div> {70,<5383.282.0>}}}]},</div>
<div> 212075,</div><div> [{{67,<0.283.0>},{member,{[],[]},212075,212075}},</div><div> {{70,<5383.282.0>},{member,{[],[]},1,1}},</div>
<div> {{73,<5066.282.0>},{member,{[],[]},0,0}}],</div><div> [<0.1271.0>],</div><div> {[],[]},</div><div> [],undefined,</div>
<div> #Fun<rabbit_misc.execute_mnesia_transaction.1>}</div><div>** Reason for termination == </div><div>** {function_clause,</div><div> [{orddict,fetch,</div><div> [{67,<0.283.0>},</div>
<div> [{{73,<5066.282.0>},</div><div> {view_member,</div><div> {73,<5066.282.0>},</div><div> [{70,<5383.282.0>}],</div><div> {73,<5066.282.0>},</div>
<div> {73,<5066.282.0>}}}]],</div><div> [{file,"orddict.erl"},{line,72}]},</div><div> {gm,check_neighbours,1,[]},</div><div> {gm,handle_info,2,[]},</div><div> {gen_server2,handle_msg,2,[]},</div>
<div> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</div><div><br></div><div>=ERROR REPORT==== 19-Feb-2014::18:34:30 ===</div><div>** Generic server <0.203.0> terminating</div><div>** Last message in was {mnesia_tm,'rabbit@FRA-VSP-32545',</div>
<div> {vote_yes,{tid,10316,<0.203.0>}}}</div><div>** When Server state == 1</div><div>** Reason for termination == </div><div>** {unexpected_info,{mnesia_tm,'rabbit@FRA-VSP-32545',</div>
<div> {vote_yes,{tid,10316,<0.203.0>}}}}</div><div><br></div><div>=ERROR REPORT==== 19-Feb-2014::18:34:30 ===</div><div>** Generic server <0.275.0> terminating</div><div>** Last message in was {'DOWN',#Ref<0.0.1.38240>,process,<5383.274.0>,</div>
<div> noconnection}</div><div>** When Server state == {state,</div><div> {70,<0.275.0>},</div><div> {{76,<5066.274.0>},#Ref<0.0.1.42305>},</div>
<div> {{73,<5383.274.0>},#Ref<0.0.1.38240>},</div><div> {resource,<<"/IEC">>,queue,</div><div> <<"activity.queue.dead">>},</div>
<div> rabbit_mirror_queue_coordinator,</div><div> {77,</div><div> [{{70,<0.275.0>},</div><div> {view_member,</div>
<div> {70,<0.275.0>},</div><div> [],</div><div> {76,<5066.274.0>},</div><div> {73,<5383.274.0>}}},</div>
<div> {{73,<5383.274.0>},</div><div> {view_member,</div><div> {73,<5383.274.0>},</div><div> [],</div>
<div> {70,<0.275.0>},</div><div> {76,<5066.274.0>}}},</div><div> {{76,<5066.274.0>},</div><div> {view_member,</div>
<div> {76,<5066.274.0>},</div><div> [],</div><div> {73,<5383.274.0>},</div><div> {70,<0.275.0>}}}]},</div>
<div> 6,</div><div> [{{70,<0.275.0>},{member,{[],[]},6,6}},</div><div> {{73,<5383.274.0>},{member,{[],[]},1,1}},</div><div> {{76,<5066.274.0>},{member,{[],[]},0,0}}],</div>
<div> [<0.1273.0>],</div><div> {[],[]},</div><div> [],undefined,</div><div> #Fun<rabbit_misc.execute_mnesia_transaction.1>}</div>
<div>** Reason for termination == </div><div>** {noproc,{gen_server2,call,</div><div> [<0.203.0>,</div><div> {submit,#Fun<rabbit_misc.6.116010224>},</div><div> infinity]}}</div>
<div><br></div><div>=ERROR REPORT==== 19-Feb-2014::18:34:30 ===</div><div>** Generic server <0.204.0> terminating</div><div>** Last message in was {mnesia_tm,'rabbit@FRA-VSP-32545',</div><div> {vote_yes,{tid,10315,<0.204.0>}}}</div>
<div>** When Server state == 2</div><div>** Reason for termination == </div><div>** {unexpected_info,{mnesia_tm,'rabbit@FRA-VSP-32545',</div><div> {vote_yes,{tid,10315,<0.204.0>}}}}</div>
<div><br></div><div>=ERROR REPORT==== 19-Feb-2014::18:34:30 ===</div><div>** Generic server <0.1268.0> terminating</div><div>** Last message in was {'$gen_cast',{gm_deaths,[<5066.266.0>,<0.267.0>]}}</div>
<div>** When Server state == {state,</div><div> {amqqueue,</div><div> {resource,<<"/IEC">>,queue,</div><div> <<"gps.queue.dead">>},</div>
<div> true,false,none,[],<0.266.0>,</div><div> [<5066.265.0>],</div><div> [<5066.265.0>],</div><div> [{vhost,<<"/IEC">>},</div>
<div> {name,<<"Queue HA">>},</div><div> {pattern,<<".queue">>},</div><div> {'apply-to',<<"queues">>},</div>
<div> {definition,</div><div> [{<<"ha-mode">>,<<"all">>},</div><div> {<<"ha-sync-mode">>,<<"automatic">>}]},</div>
<div> {priority,0}],</div><div> [{<5066.266.0>,<5066.265.0>},</div><div> {<5383.266.0>,<5383.265.0>}],</div>
<div> []},</div><div> <0.267.0>,</div><div> {state,</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> [],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> [],[],[]}}},</div><div> erlang},</div>
<div> #Fun<rabbit_mirror_queue_master.5.69128381>,</div><div> #Fun<rabbit_mirror_queue_master.6.50493311>}</div><div>** Reason for termination == </div><div>
** {{case_clause,{ok,<5066.265.0>,[]}},</div><div> [{rabbit_mirror_queue_coordinator,handle_cast,2,[]},</div><div> {gen_server2,handle_msg,2,[]},</div><div> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</div>
<div><br></div><div>=ERROR REPORT==== 19-Feb-2014::18:34:31 ===</div><div>** Generic server <0.266.0> terminating</div><div>** Last message in was {'EXIT',<0.1268.0>,</div><div> {{case_clause,{ok,<5066.265.0>,[]}},</div>
<div> [{rabbit_mirror_queue_coordinator,handle_cast,2,</div><div> []},</div><div> {gen_server2,handle_msg,2,[]},</div><div> {proc_lib,wake_up,3,</div>
<div> [{file,"proc_lib.erl"},{line,249}]}]}}</div><div>** When Server state == {q,</div><div> {amqqueue,</div><div> {resource,<<"/IEC">>,queue,<<"gps.queue.dead">>},</div>
<div> true,false,none,[],<0.266.0>,</div><div> [<5383.265.0>,<5066.265.0>],</div><div> [<5066.265.0>,<5383.265.0>],</div>
<div> [{vhost,<<"/IEC">>},</div><div> {name,<<"Queue HA">>},</div><div> {pattern,<<".queue">>},</div>
<div> {'apply-to',<<"queues">>},</div><div> {definition,</div><div> [{<<"ha-mode">>,<<"all">>},</div>
<div> {<<"ha-sync-mode">>,<<"automatic">>}]},</div><div> {priority,0}],</div><div> [{<5066.266.0>,<5066.265.0>},</div>
<div> {<5383.266.0>,<5383.265.0>},</div><div> {<0.267.0>,<0.266.0>}],</div><div> []},</div><div> none,false,rabbit_mirror_queue_master,</div>
<div> {state,</div><div> {resource,<<"/IEC">>,queue,<<"gps.queue.dead">>},</div><div> <0.267.0>,<0.1268.0>,rabbit_variable_queue,</div>
<div> {vqstate,</div><div> {0,{[],[]}},</div><div> {0,{[],[]}},</div><div> {delta,undefined,0,undefined},</div><div> {0,{[],[]}},</div>
<div> {0,{[],[]}},</div><div> 0,</div><div> {0,nil},</div><div> {0,nil},</div><div> {qistate,</div>
<div> "d:/tools/RabbitMQ Server/data/db/rabbit@FRA-VSP-32596-mnesia/queues/6IXYXKMC8M51EEAXH5MKLR0Q4",</div><div> {{dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> []},</div>
<div> undefined,0,65536,</div><div> #Fun<rabbit_variable_queue.2.81334491>,</div><div> {0,nil}},</div><div> {{client_msstate,msg_store_persistent,</div>
<div> <<55,209,140,132,77,86,75,214,37,255,72,56,103,92,</div><div> 154,75>>,</div><div> {dict,0,16,16,8,80,48,</div><div>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> {state,340043,</div>
<div> "d:/tools/RabbitMQ Server/data/db/rabbit@FRA-VSP-32596-mnesia/msg_store_persistent"},</div><div> rabbit_msg_store_ets_index,</div><div> "d:/tools/RabbitMQ Server/data/db/rabbit@FRA-VSP-32596-mnesia/msg_store_persistent",</div>
<div> <0.255.0>,344140,335946,348237,352334},</div><div> {client_msstate,msg_store_transient,</div><div> <<148,176,200,245,252,25,203,27,190,186,25,104,</div>
<div> 217,230,131,35>>,</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> {state,319558,</div><div> "d:/tools/RabbitMQ Server/data/db/rabbit@FRA-VSP-32596-mnesia/msg_store_transient"},</div>
<div> rabbit_msg_store_ets_index,</div><div> "d:/tools/RabbitMQ Server/data/db/rabbit@FRA-VSP-32596-mnesia/msg_store_transient",</div><div> <0.250.0>,323655,315461,327752,331849}},</div>
<div> true,0,0,0,infinity,0,0,0,0,0,</div><div> {rates,</div><div> {{1392,831016,530070},0},</div><div> {{1392,831016,530070},0},</div>
<div> 0.0,0.0,</div><div> {1392,831128,748070}},</div><div> {0,nil},</div><div> {0,nil},</div><div> {0,nil},</div>
<div> {0,nil},</div><div> 0,0,</div><div> {rates,</div><div> {{1392,831016,530070},0},</div><div> {{1392,831016,530070},0},</div>
<div> 0.0,0.0,</div><div> {1392,831128,748070}}},</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> [],</div><div> {set,0,16,16,8,80,48,</div>
<div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}}},</div>
<div>
{queue,[],[],0},</div><div> undefined,undefined,undefined,undefined,</div><div> {state,fine,5000,undefined},</div><div> {0,nil},</div>
<div> undefined,undefined,undefined,</div><div> {state,</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> delegate},</div><div> undefined,undefined,undefined,4,running}</div>
<div>** Reason for termination == </div><div>** {{case_clause,{ok,<5066.265.0>,[]}},</div><div> [{rabbit_mirror_queue_coordinator,handle_cast,2,[]},</div><div> {gen_server2,handle_msg,2,[]},</div><div> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</div>
</div><div><br></div><div><br></div><div>Any ideas?</div><div><br></div><div>Best regards, <br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><strong><p style="margin:0px;line-height:12pt;display:inline!important">
<strong><span style="font-size:8pt;font-family:Verdana;color:rgb(95,95,95)">________________________________________________________________</span></strong></p></strong></span><div><div><span style="font-size:x-small"><b><font color="#666666">Michaël OULLION</font></b></span></div>
<div><font color="#666666" face="verdana, sans-serif" size="1"><i>Architecte JAVA</i></font></div><div><b><font color="#FF0000"><font face="verdana, sans-serif"><img src="http://intranet2008.norbert-dentressangle.com/Intranet/Portail.nsf/3739e8870e42c9a5c12572dd004ebe9c/8cc1800a2b414d59c12573c6003af585/Contenu/0.8A6?OpenElement&FieldElemFormat=gif" width="96" height="26"></font></font></b></div>
<div><span style="font-family:verdana,sans-serif"><font color="#FF0000"><b>ND Informatique</b></font></span></div><div><span style="font-family:verdana,sans-serif"><font color="#FF0000"><b></b></font></span><span style="color:rgb(102,102,102);font-family:verdana,sans-serif;font-size:x-small">Adresse (1208 route des Pierrelles B.P. 98 BEAUSEMBLANT - 26240 Beausemblant - FRANCE)</span></div>
<div><font face="verdana, sans-serif"><span style="font-size:x-small"><font color="#666666">Tel. <a href="tel:%2B33%20%280%294%2075%2023%2068%2007" value="+33475236807" target="_blank">+33 (0)4 75 23 68 07</a></font></span></font></div>
<div><font face="verdana, sans-serif"><span style="font-size:x-small"><font color="#666666">Visit our web site at <a href="http://www.norbert-dentressangle.com/" target="_blank">www.norbert-dentressangle.com</a></font></span></font></div>
</div></div><div><font face="verdana, sans-serif"><span style="font-size:x-small"><br></span></font></div></div>
</div></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>