<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META http-equiv=Content-Type content="text/html; charset=ISO-8859-1"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hello,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We have a two-node RabbitMQ cluster running on RabbitMQ 3.2.2, Erlang R16B03-1 on Windows servers. We’ve had multiple issues over the past few days:<o:p></o:p></p><p class=MsoNormal>Sunday, 6/15/2014 – Network partition because of some planned network upgrade; restarted both nodes in sequence<o:p></o:p></p><p class=MsoNormal>Monday, 6/16/2014 – Rouge client created 5000+ concurrent connections to Rabbit; restarted both nodes in sequence and eventually tracked down and killed client<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This question is primarily about the problem that occurred on Monday. After the first node was restarted, the following messages were logged:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>=INFO REPORT==== 16-Jun-2014::17:00:47 ===<o:p></o:p></p><p class=MsoNormal>Starting RabbitMQ 3.2.2 on Erlang R16B03-1<o:p></o:p></p><p class=MsoNormal>Copyright (C) 2007-2013 GoPivotal, Inc.<o:p></o:p></p><p class=MsoNormal>Licensed under the MPL. See http://www.rabbitmq.com/<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>…<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>=ERROR REPORT==== 16-Jun-2014::17:00:49 ===<o:p></o:p></p><p class=MsoNormal>Discarding message {'$gen_call',{<0.254.0>,#Ref<0.0.0.1304>},{add_on_right,{56,<0.254.0>}}} from <0.254.0> to <0.16165.62> in an old incarnation (1) of this node (2)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>=ERROR REPORT==== 16-Jun-2014::17:00:49 ===<o:p></o:p></p><p class=MsoNormal>Discarding message {'$gen_call',{<0.256.0>,#Ref<0.0.0.1311>},{add_on_right,{56,<0.256.0>}}} from <0.256.0> to <0.16138.62> in an old incarnation (1) of this node (2)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>=ERROR REPORT==== 16-Jun-2014::17:00:49 ===<o:p></o:p></p><p class=MsoNormal>Discarding message {'$gen_call',{<0.258.0>,#Ref<0.0.0.1511>},{add_on_right,{56,<0.258.0>}}} from <0.258.0> to <0.16190.62> in an old incarnation (1) of this node (2)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>=ERROR REPORT==== 16-Jun-2014::17:00:49 ===<o:p></o:p></p><p class=MsoNormal>Discarding message {'$gen_call',{<0.260.0>,#Ref<0.0.0.1536>},{add_on_right,{29,<0.260.0>}}} from <0.260.0> to <0.16200.62> in an old incarnation (1) of this node (2)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>=INFO REPORT==== 16-Jun-2014::17:00:49 ===<o:p></o:p></p><p class=MsoNormal>Adding mirror of queue 'SalesCenter.BPBilling_AccountDelinquent' in vhost 'eventing' on node 'rabbit@SHO-P-EVGAPP-02': <5578.16037.65><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>=ERROR REPORT==== 16-Jun-2014::17:00:49 ===<o:p></o:p></p><p class=MsoNormal>** Generic server <0.250.0> terminating<o:p></o:p></p><p class=MsoNormal>** Last message in was {init,<0.185.0>}<o:p></o:p></p><p class=MsoNormal>** When Server state == {q,{amqqueue,<o:p></o:p></p><p class=MsoNormal> {resource,<<"eventing">>,queue,<o:p></o:p></p><p class=MsoNormal> <<"SalesCenter.AMS_ErrorsAndOmissions30DayExpiration">>},<o:p></o:p></p><p class=MsoNormal> true,false,none,<o:p></o:p></p><p class=MsoNormal> [{<<"x-dead-letter-exchange">>,longstr,<o:p></o:p></p><p class=MsoNormal> <<"EnterpriseEventDeadLetter">>},<o:p></o:p></p><p class=MsoNormal> {<<"x-dead-letter-routing-key">>,longstr,<o:p></o:p></p><p class=MsoNormal> <<"SalesCenter.AMS_ErrorsAndOmissions30DayExpiration">>}],<o:p></o:p></p><p class=MsoNormal> <0.250.0>,[],[],<o:p></o:p></p><p class=MsoNormal> [{vhost,<<"eventing">>},<o:p></o:p></p><p class=MsoNormal> {name,<<"ha-all">>},<o:p></o:p></p><p class=MsoNormal> {pattern,<<"^.*">>},<o:p></o:p></p><p class=MsoNormal> {'apply-to',<<"queues">>},<o:p></o:p></p><p class=MsoNormal> {definition,<o:p></o:p></p><p class=MsoNormal> [{<<"ha-mode">>,<<"all">>},<o:p></o:p></p><p class=MsoNormal> {<<"ha-sync-mode">>,<<"automatic">>}]},<o:p></o:p></p><p class=MsoNormal> {priority,0}],<o:p></o:p></p><p class=MsoNormal> [],[]},<o:p></o:p></p><p class=MsoNormal> none,false,undefined,undefined,<o:p></o:p></p><p class=MsoNormal> {queue,[],[],0},<o:p></o:p></p><p class=MsoNormal> undefined,undefined,undefined,undefined,<o:p></o:p></p><p class=MsoNormal> {state,fine,5000,undefined},<o:p></o:p></p><p class=MsoNormal> {0,nil},<o:p></o:p></p><p class=MsoNormal> undefined,undefined,undefined,<o:p></o:p></p><p class=MsoNormal> {state,<o:p></o:p></p><p class=MsoNormal> {dict,0,16,16,8,80,48,<o:p></o:p></p><p class=MsoNormal> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],<o:p></o:p></p><p class=MsoNormal> [],[]},<o:p></o:p></p><p class=MsoNormal> {{[],[],[],[],[],[],[],[],[],[],[],[],[],<o:p></o:p></p><p class=MsoNormal> [],[],[]}}},<o:p></o:p></p><p class=MsoNormal> delegate},<o:p></o:p></p><p class=MsoNormal> undefined,undefined,undefined,0,running}<o:p></o:p></p><p class=MsoNormal>** Reason for termination == <o:p></o:p></p><p class=MsoNormal>** {{function_clause,<o:p></o:p></p><p class=MsoNormal> [{rabbit_mirror_queue_slave,terminate,<o:p></o:p></p><p class=MsoNormal> [{{badmatch,true},<o:p></o:p></p><p class=MsoNormal> [{rabbit_queue_index,init,2,[]},<o:p></o:p></p><p class=MsoNormal> {rabbit_variable_queue,init,5,[]},<o:p></o:p></p><p class=MsoNormal> {rabbit_mirror_queue_slave,handle_go,1,[]},<o:p></o:p></p><p class=MsoNormal> {rabbit_mirror_queue_slave,handle_call,3,[]},<o:p></o:p></p><p class=MsoNormal> {gen_server2,handle_msg,2,[]},<o:p></o:p></p><p class=MsoNormal> {proc_lib,init_p_do_apply,3,<o:p></o:p></p><p class=MsoNormal> [{file,"proc_lib.erl"},{line,239}]}]},<o:p></o:p></p><p class=MsoNormal> {not_started,<o:p></o:p></p><p class=MsoNormal> {amqqueue,<o:p></o:p></p><p class=MsoNormal> {resource,<<"eventing">>,queue,<o:p></o:p></p><p class=MsoNormal> <<"SalesCenter.AMS_ErrorsAndOmissions30DayExpiration">>},<o:p></o:p></p><p class=MsoNormal> true,false,none,<o:p></o:p></p><p class=MsoNormal> [{<<"x-dead-letter-exchange">>,longstr,<o:p></o:p></p><p class=MsoNormal> <<"EnterpriseEventDeadLetter">>},<o:p></o:p></p><p class=MsoNormal> {<<"x-dead-letter-routing-key">>,longstr,<o:p></o:p></p><p class=MsoNormal> <<"SalesCenter.AMS_ErrorsAndOmissions30DayExpiration">>}],<o:p></o:p></p><p class=MsoNormal> <0.250.0>,[],[],<o:p></o:p></p><p class=MsoNormal> [{vhost,<<"eventing">>},<o:p></o:p></p><p class=MsoNormal> {name,<<"ha-all">>},<o:p></o:p></p><p class=MsoNormal> {pattern,<<"^.*">>},<o:p></o:p></p><p class=MsoNormal> {'apply-to',<<"queues">>},<o:p></o:p></p><p class=MsoNormal> {definition,<o:p></o:p></p><p class=MsoNormal> [{<<"ha-mode">>,<<"all">>},<o:p></o:p></p><p class=MsoNormal> {<<"ha-sync-mode">>,<<"automatic">>}]},<o:p></o:p></p><p class=MsoNormal> {priority,0}],<o:p></o:p></p><p class=MsoNormal> [{<0.256.0>,<0.250.0>}],<o:p></o:p></p><p class=MsoNormal> []}}],<o:p></o:p></p><p class=MsoNormal> []},<o:p></o:p></p><p class=MsoNormal> {gen_server2,terminate,3,[]},<o:p></o:p></p><p class=MsoNormal> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},<o:p></o:p></p><p class=MsoNormal> {gen_server2,call,[<5578.15838.65>,go,infinity]}}<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>After this, when going to the queue page (/#/queues/eventing/SalesCenter.AMS_ErrorsAndOmissions30DayExpiration) in the management console, I get TypeError: Cannot read property 'ram_msg_count' of undefined. On the list of queues, this queue is listed as “Active” (most queues are listed as “Idle”) and the Read/Unacked/Total messages for this queue shows question marks. The only way we were able to resolve was to stop the cluster entirely and start it back up.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any thoughts on what went wrong, or how to avoid it in the future?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal>Andrew Miller<o:p></o:p></p></div>
<p>
This e-mail is confidential.� If you are not the intended recipient, you must not disclose or use the information contained in it.� If you have received this e-mail in error, please tell us immediately by return e-mail and delete the document. No recipient may use the information in this e-mail in violation of any civil or criminal statute. Sentry disclaims all liability for any unauthorized uses of this e-mail or its contents. Sentry accepts no liability or responsibility for any damage caused by any virus transmitted with this e-mail.</p><p />
</body></html>