[rabbitmq-discuss] Queues disappeared after a cluster upgrade to 3.1.5, Erlang R16B01
Jason McIntosh
mcintoshj at gmail.com
Fri Aug 23 22:51:22 BST 2013
Here's some other details from the logs if it might help (note I discovered
the web-gui issue was a cache issue):
=INFO REPORT==== 23-Aug-2013::15:37:43 ===
upgrades: Mnesia dir backed up to
"/data/rabbitmq/rabbitmq/mnesia/cluster-upgrade-backup"
=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: 1 to apply
=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: Breaking cluster
=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: Applying rabbit_upgrade_functions:exchange_decorators
=INFO REPORT==== 23-Aug-2013::15:37:43 ===
mnesia upgrades: All upgrades applied successfully
=INFO REPORT==== 23-Aug-2013::15:37:44 ===
Starting RabbitMQ 3.1.5 on Erlang R16B01
Copyright (C) 2007-2013 GoPivotal, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
=INFO REPORT==== 23-Aug-2013::15:37:44 ===
node : cluster at rabbitmqm10t
home dir : /var/lib/rabbitmq
config file(s) : (none)
cookie hash : 9fmMn2E9ImSXspwK+tgGgQ==
log : /data/rabbitmq/cluster.log
sasl log : /data/rabbitmq/cluster-sasl.log
database dir : /data/rabbitmq/rabbitmq/mnesia/cluster
=INFO REPORT==== 23-Aug-2013::15:37:45 ===
Limiting to approx 924 file handles (829 sockets)
=INFO REPORT==== 23-Aug-2013::15:37:45 ===
upgrades: Mnesia backup removed
=INFO REPORT==== 23-Aug-2013::15:37:45 ===
Memory limit set to 396MB of 991MB total.
=INFO REPORT==== 23-Aug-2013::15:37:45 ===
Disk free limit set to 1000MB
On Fri, Aug 23, 2013 at 4:21 PM, Jason McIntosh <mcintoshj at gmail.com> wrote:
> We just upgraded our Alpha environment to 3.1.5 and Erlang R16B01. In the
> process, ALL of our queue definitions disappeared. Checking logs, I get a
> whole lot of errors like the below (names redacted). Thankfully, I can
> auto-generate all the stuff again, but I'm having errors on my "upgrader"
> node right now that is preventing it from even working - the webgui gives a
> "ReferenceError: queue_length is not defined" error and doesn't display
> any information. Note, I followed the instructions - picked an upgrader
> node, shutdown all the other nodes in order, brought the "upgrader" node
> back online, then the others in sequential order. This has me really
> really spooked though.
>
> Any advice??
> Thanks!
> Jason
>
> =ERROR REPORT==== 23-Aug-2013::15:37:50 ===
> ** Generic server <0.303.0> terminating
> ** Last message in was {'EXIT',<0.350.0>,normal}
> ** When Server state == {q,
> {amqqueue,
> {resource,<<"EXCHANGE">>,queue,
> <<"QUEUE">>},
> true,false,none,
> [{<<"x-ha-policy">>,longstr,<<"all">>}],
> <0.303.0>,[],[],
> [{vhost,<<"pca">>},
> {name,<<"HA">>},
> {pattern,<<"^(?!amq\\.).*">>},
> {definition,[{<<"ha-mode">>,<<"all">>}]},
> {priority,0}],
> [{<0.261.0>,<0.260.0>},
> {<0.308.0>,<0.260.0>},
> {<0.299.0>,<0.258.0>},
> {<0.562.0>,<0.561.0>}]},
> none,false,rabbit_mirror_queue_master,
> {state,
> {resource,<<"EXCHANGE">>,queue,
> <<"QUEUE">>},
> <0.351.0>,<0.350.0>,rabbit_variable_queue,
> {vqstate,
> {0,{[],[]}},
> {0,{[],[]}},
> {delta,0,0,0},
> {0,{[],[]}},
> {0,{[],[]}},
> 0,
> {0,nil},
> {0,nil},
> {qistate,
>
> "/data/rabbitmq/rabbitmq/mnesia/cluster/queues/8FHU53A729K2C3KLMS8XPVKJD",
> {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> []},
> #Ref<0.0.0.1702>,0,65536,
> #Fun<rabbit_variable_queue.2.81334491>,
> {0,nil}},
> {{client_msstate,msg_store_persistent,
>
> <<247,77,14,2,221,67,18,168,38,83,171,47,207,31,
> 187,253>>,
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> {state,450623,
>
> "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent"},
> rabbit_msg_store_ets_index,
>
> "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent",
> <0.301.0>,454733,446535,458830,462927},
> {client_msstate,msg_store_transient,
>
> <<12,4,24,179,230,120,71,131,21,190,238,120,9,176,
> 231,238>>,
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> undefined,undefined,undefined,running}
> ** Reason for termination ==
> ** {{badmatch,{error,not_found}},
> [{rabbit_mirror_queue_master,stop_all_slaves,2,[]},
> {rabbit_mirror_queue_master,delete_and_terminate,2,[]},
> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
> {rabbit_amqqueue_process,terminate_shutdown,2,[]},
> {gen_server2,terminate,3,[]},
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
> ** In 'terminate' callback with reason ==
> ** normal
>
>
> --
> Jason McIntosh
> http://mcintosh.poetshome.com/blog/
> 573-424-7612
>
--
Jason McIntosh
http://mcintosh.poetshome.com/blog/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130823/00bb7427/attachment.htm>
More information about the rabbitmq-discuss
mailing list