[rabbitmq-discuss] Queues disappeared after a cluster upgrade to 3.1.5, Erlang R16B01

Fri Aug 23 22:51:22 BST 2013

Here's some other details from the logs if it might help (note I discovered
the web-gui issue was a cache issue):

=INFO REPORT==== 23-Aug-2013::15:37:43 ===
upgrades: Mnesia dir backed up to
"/data/rabbitmq/rabbitmq/mnesia/cluster-upgrade-backup"

=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: 1 to apply

=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: Breaking cluster

=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: Applying rabbit_upgrade_functions:exchange_decorators

=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: All upgrades applied successfully

=INFO REPORT==== 23-Aug-2013::15:37:44 ===
Starting RabbitMQ 3.1.5 on Erlang R16B01
Copyright (C) 2007-2013 GoPivotal, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 23-Aug-2013::15:37:44 ===
node           : cluster at rabbitmqm10t
home dir       : /var/lib/rabbitmq
config file(s) : (none)
cookie hash    : 9fmMn2E9ImSXspwK+tgGgQ==
log            : /data/rabbitmq/cluster.log
sasl log       : /data/rabbitmq/cluster-sasl.log
database dir   : /data/rabbitmq/rabbitmq/mnesia/cluster

=INFO REPORT==== 23-Aug-2013::15:37:45 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 23-Aug-2013::15:37:45 ===
upgrades: Mnesia backup removed

=INFO REPORT==== 23-Aug-2013::15:37:45 ===
Memory limit set to 396MB of 991MB total.

=INFO REPORT==== 23-Aug-2013::15:37:45 ===
Disk free limit set to 1000MB

On Fri, Aug 23, 2013 at 4:21 PM, Jason McIntosh <mcintoshj at gmail.com> wrote:

> We just upgraded our Alpha environment to 3.1.5 and Erlang R16B01.  In the
> process, ALL of our queue definitions disappeared.  Checking logs, I get a
>  whole lot of errors like the below (names redacted).  Thankfully, I can
> auto-generate all the stuff again, but I'm having errors on my "upgrader"
> node right now that is preventing it from even working - the webgui gives a
> "ReferenceError: queue_length is not defined" error and doesn't display
> any information. Note, I followed the instructions - picked an upgrader
> node, shutdown all the other nodes in order, brought the "upgrader" node
> back online, then the others in sequential order.  This has me really
> really spooked though.
>
> Any advice??
> Thanks!
> Jason
>
> =ERROR REPORT==== 23-Aug-2013::15:37:50 ===
> ** Generic server <0.303.0> terminating
> ** Last message in was {'EXIT',<0.350.0>,normal}
> ** When Server state == {q,
>                          {amqqueue,
>                           {resource,<<"EXCHANGE">>,queue,
>                            <<"QUEUE">>},
>                           true,false,none,
>                           [{<<"x-ha-policy">>,longstr,<<"all">>}],
>                           <0.303.0>,[],[],
>                           [{vhost,<<"pca">>},
>                            {name,<<"HA">>},
>                            {pattern,<<"^(?!amq\\.).*">>},
>                            {definition,[{<<"ha-mode">>,<<"all">>}]},
>                            {priority,0}],
>                           [{<0.261.0>,<0.260.0>},
>                            {<0.308.0>,<0.260.0>},
>                            {<0.299.0>,<0.258.0>},
>                            {<0.562.0>,<0.561.0>}]},
>                          none,false,rabbit_mirror_queue_master,
>                          {state,
>                           {resource,<<"EXCHANGE">>,queue,
>                            <<"QUEUE">>},
>                           <0.351.0>,<0.350.0>,rabbit_variable_queue,
>                           {vqstate,
>                            {0,{[],[]}},
>                            {0,{[],[]}},
>                            {delta,0,0,0},
>                            {0,{[],[]}},
>                            {0,{[],[]}},
>                            0,
>                            {0,nil},
>                            {0,nil},
>                            {qistate,
>
> "/data/rabbitmq/rabbitmq/mnesia/cluster/queues/8FHU53A729K2C3KLMS8XPVKJD",
>                             {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []}}},
>                              []},
>                             #Ref<0.0.0.1702>,0,65536,
>                             #Fun<rabbit_variable_queue.2.81334491>,
>                             {0,nil}},
>                            {{client_msstate,msg_store_persistent,
>
>  <<247,77,14,2,221,67,18,168,38,83,171,47,207,31,
>                                187,253>>,
>                              {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []}}},
>                              {state,450623,
>
> "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent"},
>                              rabbit_msg_store_ets_index,
>
>  "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent",
>                              <0.301.0>,454733,446535,458830,462927},
>                             {client_msstate,msg_store_transient,
>
>  <<12,4,24,179,230,120,71,131,21,190,238,120,9,176,
>                                231,238>>,
>                              {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []}}},
>                         undefined,undefined,undefined,running}
> ** Reason for termination ==
> ** {{badmatch,{error,not_found}},
>     [{rabbit_mirror_queue_master,stop_all_slaves,2,[]},
>      {rabbit_mirror_queue_master,delete_and_terminate,2,[]},
>      {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
>      {rabbit_amqqueue_process,terminate_shutdown,2,[]},
>      {gen_server2,terminate,3,[]},
>      {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
> ** In 'terminate' callback with reason ==
> ** normal
>
>
> --
> Jason McIntosh
> http://mcintosh.poetshome.com/blog/
> 573-424-7612
>

-- 
Jason McIntosh
http://mcintosh.poetshome.com/blog/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130823/00bb7427/attachment.htm>