<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style>
<!--
@font-face
{font-family:Calibri}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.MsoChpDefault
{font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.WordSection1
{}
-->
</style>
</head>
<body lang="EN-AU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">While testing RabbitMQ version 2.8.7 I've encountered an issue where the slave node in a HA pair terminates when the master is forcefully rebooted.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The environment is Centos 6.3 with RabbitMQ 2.8.7, Erlang R14B, with a durable queue mirrored across a pair of nodes in a HA cluster. The master is rebooted using reboot -nf to simulate an ungraceful failure. When the master comes back
up the slave node terminates with error log shown below.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The behaviour when testing with version 2.8.6 was that the queue would vanish from the cluster. With version 2.8.7 the queue remains intact, but the slave node terminates.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I've found that if the rabbitmq-server is not configured to start at boot the slave node eventually promotes itself to master and survives the eventual restart of the original master.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Nathanael</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 5-Oct-2012::17:24:35 ===</p>
<p class="MsoNormal">rabbit on node 'rabbit@zg-dev-nr-002' down</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 5-Oct-2012::17:24:35 ===</p>
<p class="MsoNormal">Statistics database started.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 5-Oct-2012::17:24:35 ===</p>
<p class="MsoNormal">Mirrored-queue (queue 'zg-ha-test' in vhost '/'): Slave</p>
<p class="MsoNormal"><'rabbit@zg-dev-nr-001'.3.252.0> saw deaths of mirrors</p>
<p class="MsoNormal"><'rabbit@zg-dev-nr-002'.2.252.0></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 5-Oct-2012::17:24:35 ===</p>
<p class="MsoNormal">Mirrored-queue (queue 'zg-ha-test' in vhost '/'): Promoting slave</p>
<p class="MsoNormal"><'rabbit@zg-dev-nr-001'.3.252.0> to master</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 5-Oct-2012::17:24:35 ===</p>
<p class="MsoNormal">** Generic server <0.253.0> terminating</p>
<p class="MsoNormal">** Last message in was {mnesia_tm,'rabbit@zg-dev-nr-002',</p>
<p class="MsoNormal"> {vote_yes,{tid,2235,<0.253.0>}}}</p>
<p class="MsoNormal">** When Server state == {state,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {resource,<<"/">>,queue,<<"zg-ha-test">>},</p>
<p class="MsoNormal"> rabbit_mirror_queue_coordinator,</p>
<p class="MsoNormal"> {6,</p>
<p class="MsoNormal"> [{{3,<0.253.0>},</p>
<p class="MsoNormal"> {view_member,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> [],</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {3,<0.253.0>}}}]},</p>
<p class="MsoNormal"> 0,[],</p>
<p class="MsoNormal"> [<0.336.0>],</p>
<p class="MsoNormal"> {[],[]},</p>
<p class="MsoNormal"> [],undefined}</p>
<p class="MsoNormal">** Reason for termination ==</p>
<p class="MsoNormal">** {function_clause,</p>
<p class="MsoNormal"> [{gm,handle_info,</p>
<p class="MsoNormal"> [{mnesia_tm,'rabbit@zg-dev-nr-002',</p>
<p class="MsoNormal"> {vote_yes,{tid,2235,<0.253.0>}}},</p>
<p class="MsoNormal"> {state,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {resource,<<"/">>,queue,<<"zg-ha-test">>},</p>
<p class="MsoNormal"> rabbit_mirror_queue_coordinator,</p>
<p class="MsoNormal"> {6,</p>
<p class="MsoNormal"> [{{3,<0.253.0>},</p>
<p class="MsoNormal"> {view_member,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> [],</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {3,<0.253.0>}}}]},</p>
<p class="MsoNormal"> 0,[],</p>
<p class="MsoNormal"> [<0.336.0>],</p>
<p class="MsoNormal"> {[],[]},</p>
<p class="MsoNormal"> [],undefined}]},</p>
<p class="MsoNormal"> {gen_server2,handle_msg,2},</p>
<p class="MsoNormal"> {proc_lib,wake_up,3}]}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 5-Oct-2012::17:24:36 ===</p>
<p class="MsoNormal">** Generic server <0.252.0> terminating</p>
<p class="MsoNormal">** Last message in was {'EXIT',<0.336.0>,</p>
<p class="MsoNormal"> {function_clause,</p>
<p class="MsoNormal"> [{gm,handle_info,</p>
<p class="MsoNormal"> [{mnesia_tm,'rabbit@zg-dev-nr-002',</p>
<p class="MsoNormal"> {vote_yes,{tid,2235,<0.253.0>}}},</p>
<p class="MsoNormal"> {state,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {resource,<<"/">>,queue,<<"zg-ha-test">>},</p>
<p class="MsoNormal"> rabbit_mirror_queue_coordinator,</p>
<p class="MsoNormal"> {6,</p>
<p class="MsoNormal"> [{{3,<0.253.0>},</p>
<p class="MsoNormal"> {view_member,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> [],</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {3,<0.253.0>}}}]},</p>
<p class="MsoNormal"> 0,[],</p>
<p class="MsoNormal"> [<0.336.0>],</p>
<p class="MsoNormal"> {[],[]},</p>
<p class="MsoNormal"> [],undefined}]},</p>
<p class="MsoNormal"> {gen_server2,handle_msg,2},</p>
<p class="MsoNormal"> {proc_lib,wake_up,3}]}}</p>
<p class="MsoNormal">** When Server state == {q,</p>
<p class="MsoNormal"> {amqqueue,</p>
<p class="MsoNormal"> {resource,<<"/">>,queue,<<"zg-ha-test">>},</p>
<p class="MsoNormal"> true,false,none,</p>
<p class="MsoNormal"> [{<<"x-ha-policy">>,longstr,<<"all">>}],</p>
<p class="MsoNormal"> <0.252.0>,[],all},</p>
<p class="MsoNormal"> none,false,rabbit_mirror_queue_master,</p>
<p class="MsoNormal"> {state,<0.253.0>,<0.336.0>,rabbit_variable_queue,</p>
<p class="MsoNormal"> {vqstate,</p>
<p class="MsoNormal"> {0,{[],[]}},</p>
<p class="MsoNormal"> {0,{[],[]}},</p>
<p class="MsoNormal"> {delta,undefined,0,undefined},</p>
<p class="MsoNormal"> {0,{[],[]}},</p>
<p class="MsoNormal">[... lots of stuff omitted...]</p>
<p class="MsoNormal"> {291,</p>
<p class="MsoNormal"> 29148,</p>
<p class="MsoNormal"> {0,nil},</p>
<p class="MsoNormal"> undefined,</p>
<p class="MsoNormal"> {0,nil},</p>
<p class="MsoNormal"> {qistate,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">"/var/lib/rabbitmq/mnesia/rabbit@zg-dev-nr-001/queues/EVSP28XGBN529KMXDI7YW0GM2",</p>
<p class="MsoNormal"> {{dict,0,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []},</p>
<p class="MsoNormal"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []}}},</p>
<p class="MsoNormal"> []},</p>
<p class="MsoNormal"> undefined,0,262144,</p>
<p class="MsoNormal"> #Fun<rabbit_variable_queue.2.121033067>,</p>
<p class="MsoNormal"> {0,nil}},</p>
<p class="MsoNormal"> {{client_msstate,msg_store_persistent,</p>
<p class="MsoNormal"> <<64,124,193,190,179,5,132,252,0,55,122,227,216,</p>
<p class="MsoNormal"> 123,112,237>>,</p>
<p class="MsoNormal"> {dict,0,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []},</p>
<p class="MsoNormal"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []}}},</p>
<p class="MsoNormal"> {state,303177,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">"/var/lib/rabbitmq/mnesia/rabbit@zg-dev-nr-001/msg_store_persistent"},</p>
<p class="MsoNormal"> rabbit_msg_store_ets_index,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">"/var/lib/rabbitmq/mnesia/rabbit@zg-dev-nr-001/msg_store_persistent",</p>
<p class="MsoNormal"> <0.246.0>,307274,299080,311371,315468},</p>
<p class="MsoNormal"> {client_msstate,msg_store_transient,</p>
<p class="MsoNormal"> <<38,48,43,9,149,2,213,22,77,78,223,24,149,53,175,</p>
<p class="MsoNormal"> 245>>,</p>
<p class="MsoNormal"> {dict,0,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []},</p>
<p class="MsoNormal"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []}}},</p>
<p class="MsoNormal"> {state,282692,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">"/var/lib/rabbitmq/mnesia/rabbit@zg-dev-nr-001/msg_store_transient"},</p>
<p class="MsoNormal"> rabbit_msg_store_ets_index,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">"/var/lib/rabbitmq/mnesia/rabbit@zg-dev-nr-001/msg_store_transient",</p>
<p class="MsoNormal"> <0.241.0>,286789,278595,290886,294983}},</p>
<p class="MsoNormal"> true,0,#Fun<rabbit_mirror_queue_slave.1.40624233>,</p>
<p class="MsoNormal"> 291,0,infinity,291,0,152,0,291,</p>
<p class="MsoNormal"> {rates,</p>
<p class="MsoNormal"> {{1349,429050,591110},0},</p>
<p class="MsoNormal"> {{1349,429050,591110},0},</p>
<p class="MsoNormal"> 1076.4632093998348,1076.4632093998348,</p>
<p class="MsoNormal"> {1349,429051,592073}},</p>
<p class="MsoNormal"> {0,nil},</p>
<p class="MsoNormal"> {0,nil},</p>
<p class="MsoNormal"> {0,nil},</p>
<p class="MsoNormal"> {0,nil},</p>
<p class="MsoNormal"> 0,0,</p>
<p class="MsoNormal"> {rates,</p>
<p class="MsoNormal"> {{1349,429050,591110},0},</p>
<p class="MsoNormal"> {{1349,429050,591110},0},</p>
<p class="MsoNormal"> 1071.7981467372135,1076.4632093998348,</p>
<p class="MsoNormal"> {1349,429051,592073}}},</p>
<p class="MsoNormal"> 0,</p>
<p class="MsoNormal"> {dict,6,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</p>
<p class="MsoNormal"> {{[],[],</p>
<p class="MsoNormal"> [[<<218,200,61,58,97,50,72,197,144,158,220,215,</p>
<p class="MsoNormal"> 159,163,253,105>>|</p>
<p class="MsoNormal"> published]],</p>
<p class="MsoNormal"> [[<<80,182,167,56,95,139,134,134,192,40,123,239,</p>
<p class="MsoNormal"> 62,193,49,206>>|</p>
<p class="MsoNormal"> published],</p>
<p class="MsoNormal"> [<<159,219,2,228,110,119,150,246,97,74,183,72,</p>
<p class="MsoNormal"> 254,233,74,33>>|</p>
<p class="MsoNormal"> published],</p>
<p class="MsoNormal"> [<<223,108,153,205,69,107,192,83,254,145,181,</p>
<p class="MsoNormal"> 172,15,61,33,190>>|</p>
<p class="MsoNormal"> published]],</p>
<p class="MsoNormal"> [],[],[],[],</p>
<p class="MsoNormal"> [[<<8,84,74,65,208,81,184,115,74,86,225,237,241,</p>
<p class="MsoNormal"> 172,148,18>>|</p>
<p class="MsoNormal"> published]],</p>
<p class="MsoNormal"> [],[],[],[],[],</p>
<p class="MsoNormal"> [[<<"�jq$���S!�,a��u�">>|published]],</p>
<p class="MsoNormal"> []}}},</p>
<p class="MsoNormal"> [],</p>
<p class="MsoNormal"> {dict,0,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</p>
<p class="MsoNormal"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []}}},</p>
<p class="MsoNormal"> {set,0,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</p>
<p class="MsoNormal"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</p>
<p class="MsoNormal"> []}}}},</p>
<p class="MsoNormal"> {[],[]},</p>
<p class="MsoNormal"> undefined,undefined,#Ref<0.0.0.1342>,undefined,</p>
<p class="MsoNormal"> {state,fine,5000,#Ref<0.0.0.1406>},</p>
<p class="MsoNormal"> {0,nil},</p>
<p class="MsoNormal"> undefined,undefined,undefined,</p>
<p class="MsoNormal"> {dict,1,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</p>
<p class="MsoNormal"> {{[],[],[],[],[],[],</p>
<p class="MsoNormal"> [[<7527.317.0>|#Ref<0.0.0.732>]],</p>
<p class="MsoNormal"> [],[],[],[],[],[],[],[],[]}}},</p>
<p class="MsoNormal"> 1,</p>
<p class="MsoNormal"> {{0,nil},{0,nil}},</p>
<p class="MsoNormal"> undefined,</p>
<p class="MsoNormal"> {dict,0,16,16,8,80,48,</p>
<p class="MsoNormal"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</p>
<p class="MsoNormal"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</p>
<p class="MsoNormal"> undefined,undefined}</p>
<p class="MsoNormal">** Reason for termination ==</p>
<p class="MsoNormal">** {noproc,{gen_server2,call,[<0.253.0>,info,infinity]}}</p>
<p class="MsoNormal">** In 'terminate' callback with reason ==</p>
<p class="MsoNormal">** {function_clause,</p>
<p class="MsoNormal"> [{gm,handle_info,</p>
<p class="MsoNormal"> [{mnesia_tm,'rabbit@zg-dev-nr-002',</p>
<p class="MsoNormal"> {vote_yes,{tid,2235,<0.253.0>}}},</p>
<p class="MsoNormal"> {state,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {{3,<0.253.0>},undefined},</p>
<p class="MsoNormal"> {resource,<<"/">>,queue,<<"zg-ha-test">>},</p>
<p class="MsoNormal"> rabbit_mirror_queue_coordinator,</p>
<p class="MsoNormal"> {6,</p>
<p class="MsoNormal"> [{{3,<0.253.0>},</p>
<p class="MsoNormal"> {view_member,</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> [],</p>
<p class="MsoNormal"> {3,<0.253.0>},</p>
<p class="MsoNormal"> {3,<0.253.0>}}}]},</p>
<p class="MsoNormal"> 0,[],</p>
<p class="MsoNormal"> [<0.336.0>],</p>
<p class="MsoNormal"> {[],[]},</p>
<p class="MsoNormal"> [],undefined}]},</p>
<p class="MsoNormal"> {gen_server2,handle_msg,2},</p>
<p class="MsoNormal"> {proc_lib,wake_up,3}]}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 5-Oct-2012::17:24:36 ===</p>
<p class="MsoNormal">rabbit on node 'rabbit@zg-dev-nr-002' up</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 5-Oct-2012::17:24:36 ===</p>
<p class="MsoNormal">** Generic server <0.292.0> terminating</p>
<p class="MsoNormal">** Last message in was {mnesia_tm,'rabbit@zg-dev-nr-002',</p>
<p class="MsoNormal"> {vote_yes,{tid,2236,<0.292.0>}}}</p>
<p class="MsoNormal">** When Server state == {state,<0.290.0>,<0.291.0>,rabbit_mgmt_sup,</p>
<p class="MsoNormal"> [{rabbit_mgmt_db,</p>
<p class="MsoNormal"> {rabbit_mgmt_db,start_link,[]},</p>
<p class="MsoNormal"> permanent,4294967295,worker,</p>
<p class="MsoNormal"> [rabbit_mgmt_db]}]}</p>
<p class="MsoNormal">** Reason for termination ==</p>
<p class="MsoNormal">** {unexpected_info,{mnesia_tm,'rabbit@zg-dev-nr-002',</p>
<p class="MsoNormal"> {vote_yes,{tid,2236,<0.292.0>}}}}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 5-Oct-2012::17:24:36 ===</p>
<p class="MsoNormal"> application: rabbitmq_management</p>
<p class="MsoNormal"> exited: shutdown</p>
<p class="MsoNormal"> type: permanent</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 5-Oct-2012::17:24:36 ===</p>
<p class="MsoNormal">stopped TCP Listener on [::]:5672</p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
ZettaServe Disclaimer: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately if you have received this email by mistake and delete this email from your system. Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses.
ZettaServe Pty Ltd accepts no liability for any damage caused by any virus transmitted by this email.<br>
<br>
</font>
</body>
</html>