<div dir="ltr">We just upgraded our Alpha environment to 3.1.5 and Erlang R16B01. In the process, ALL of our queue definitions disappeared. Checking logs, I get a whole lot of errors like the below (names redacted). Thankfully, I can auto-generate all the stuff again, but I'm having errors on my "upgrader" node right now that is preventing it from even working - the webgui gives a "<span style="color:rgb(72,72,72);font-family:Verdana,sans-serif;font-size:12px">ReferenceError: queue_length is not defined" error and doesn't display any information. Note, I followed the instructions - picked an upgrader node, shutdown all the other nodes in order, brought the "upgrader" node back online, then the others in sequential order. This has me really really spooked though.</span><div>
<font color="#484848" face="Verdana, sans-serif"><span style="font-size:12px"><br></span></font></div><div><font color="#484848" face="Verdana, sans-serif"><span style="font-size:12px">Any advice??<br>Thanks!<br>Jason<br>
</span></font><div><br></div><div><div>=ERROR REPORT==== 23-Aug-2013::15:37:50 ===</div><div>** Generic server <0.303.0> terminating</div><div>** Last message in was {'EXIT',<0.350.0>,normal}</div><div>
** When Server state == {q,</div><div> {amqqueue,</div><div> {resource,<<"EXCHANGE">>,queue,</div><div> <<"QUEUE">>},</div>
<div> true,false,none,</div><div> [{<<"x-ha-policy">>,longstr,<<"all">>}],</div><div> <0.303.0>,[],[],</div>
<div> [{vhost,<<"pca">>},</div><div> {name,<<"HA">>},</div><div> {pattern,<<"^(?!amq\\.).*">>},</div>
<div> {definition,[{<<"ha-mode">>,<<"all">>}]},</div><div> {priority,0}],</div><div> [{<0.261.0>,<0.260.0>},</div>
<div> {<0.308.0>,<0.260.0>},</div><div> {<0.299.0>,<0.258.0>},</div><div> {<0.562.0>,<0.561.0>}]},</div><div>
none,false,rabbit_mirror_queue_master,</div><div> {state,</div><div> {resource,<<"EXCHANGE">>,queue,</div><div> <<"QUEUE">>},</div>
<div> <0.351.0>,<0.350.0>,rabbit_variable_queue,</div><div> {vqstate,</div><div> {0,{[],[]}},</div><div> {0,{[],[]}},</div>
<div> {delta,0,0,0},</div><div> {0,{[],[]}},</div><div> {0,{[],[]}},</div><div> 0,</div><div> {0,nil},</div>
<div> {0,nil},</div><div> {qistate,</div><div> "/data/rabbitmq/rabbitmq/mnesia/cluster/queues/8FHU53A729K2C3KLMS8XPVKJD",</div><div>
{{dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []}}},</div><div> []},</div><div> #Ref<0.0.0.1702>,0,65536,</div><div> #Fun<rabbit_variable_queue.2.81334491>,</div>
<div> {0,nil}},</div><div> {{client_msstate,msg_store_persistent,</div><div> <<247,77,14,2,221,67,18,168,38,83,171,47,207,31,</div><div>
187,253>>,</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> {state,450623,</div><div> "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent"},</div>
<div> rabbit_msg_store_ets_index,</div><div> "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent",</div><div> <0.301.0>,454733,446535,458830,462927},</div>
<div> {client_msstate,msg_store_transient,</div><div> <<12,4,24,179,230,120,71,131,21,190,238,120,9,176,</div><div> 231,238>>,</div>
<div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []}}},</div><div> undefined,undefined,undefined,running}</div><div>** Reason for termination ==</div><div>** {{badmatch,{error,not_found}},</div><div> [{rabbit_mirror_queue_master,stop_all_slaves,2,[]},</div>
<div> {rabbit_mirror_queue_master,delete_and_terminate,2,[]},</div><div> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},</div><div> {rabbit_amqqueue_process,terminate_shutdown,2,[]},</div>
<div> {gen_server2,terminate,3,[]},</div><div> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</div><div>** In 'terminate' callback with reason ==</div><div>** normal</div><div><br></div>
<div><br></div>-- <br>Jason McIntosh<br><a href="http://mcintosh.poetshome.com/blog/">http://mcintosh.poetshome.com/blog/</a><br>573-424-7612
</div></div></div>