[rabbitmq-discuss] Queues disappeared after a cluster upgrade to 3.1.5, Erlang R16B01

Fri Aug 23 22:21:37 BST 2013

We just upgraded our Alpha environment to 3.1.5 and Erlang R16B01.  In the
process, ALL of our queue definitions disappeared.  Checking logs, I get a
 whole lot of errors like the below (names redacted).  Thankfully, I can
auto-generate all the stuff again, but I'm having errors on my "upgrader"
node right now that is preventing it from even working - the webgui gives a
"ReferenceError: queue_length is not defined" error and doesn't display any
information. Note, I followed the instructions - picked an upgrader node,
shutdown all the other nodes in order, brought the "upgrader" node back
online, then the others in sequential order.  This has me really really
spooked though.

Any advice??
Thanks!
Jason

=ERROR REPORT==== 23-Aug-2013::15:37:50 ===
** Generic server <0.303.0> terminating
** Last message in was {'EXIT',<0.350.0>,normal}
** When Server state == {q,
                         {amqqueue,
                          {resource,<<"EXCHANGE">>,queue,
                           <<"QUEUE">>},
                          true,false,none,
                          [{<<"x-ha-policy">>,longstr,<<"all">>}],
                          <0.303.0>,[],[],
                          [{vhost,<<"pca">>},
                           {name,<<"HA">>},
                           {pattern,<<"^(?!amq\\.).*">>},
                           {definition,[{<<"ha-mode">>,<<"all">>}]},
                           {priority,0}],
                          [{<0.261.0>,<0.260.0>},
                           {<0.308.0>,<0.260.0>},
                           {<0.299.0>,<0.258.0>},
                           {<0.562.0>,<0.561.0>}]},
                         none,false,rabbit_mirror_queue_master,
                         {state,
                          {resource,<<"EXCHANGE">>,queue,
                           <<"QUEUE">>},
                          <0.351.0>,<0.350.0>,rabbit_variable_queue,
                          {vqstate,
                           {0,{[],[]}},
                           {0,{[],[]}},
                           {delta,0,0,0},
                           {0,{[],[]}},
                           {0,{[],[]}},
                           0,
                           {0,nil},
                           {0,nil},
                           {qistate,

"/data/rabbitmq/rabbitmq/mnesia/cluster/queues/8FHU53A729K2C3KLMS8XPVKJD",
                            {{dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             []},
                            #Ref<0.0.0.1702>,0,65536,
                            #Fun<rabbit_variable_queue.2.81334491>,
                            {0,nil}},
                           {{client_msstate,msg_store_persistent,

 <<247,77,14,2,221,67,18,168,38,83,171,47,207,31,
                               187,253>>,
                             {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             {state,450623,

"/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent"},
                             rabbit_msg_store_ets_index,

 "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent",
                             <0.301.0>,454733,446535,458830,462927},
                            {client_msstate,msg_store_transient,

 <<12,4,24,179,230,120,71,131,21,190,238,120,9,176,
                               231,238>>,
                             {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                        undefined,undefined,undefined,running}
** Reason for termination ==
** {{badmatch,{error,not_found}},
    [{rabbit_mirror_queue_master,stop_all_slaves,2,[]},
     {rabbit_mirror_queue_master,delete_and_terminate,2,[]},
     {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
     {rabbit_amqqueue_process,terminate_shutdown,2,[]},
     {gen_server2,terminate,3,[]},
     {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
** In 'terminate' callback with reason ==
** normal

-- 
Jason McIntosh
http://mcintosh.poetshome.com/blog/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130823/7bf3e60e/attachment.htm>