[rabbitmq-discuss] Queues disappeared after a cluster upgrade to 3.1.5, Erlang R16B01

Jason McIntosh mcintoshj at gmail.com
Fri Aug 23 23:01:59 BST 2013


More investigation...  R16B01 installs here:
/usr/lib/erlang/erts-5.10.2/

A ps auf shows /usr/lib/erlang/erts-5.9.3.1/bin/epmd -daemon as still
running.  SO I'm wondering if that might have an impact.  Note here's how
I'm doing my install:

stop rabbit on server X (upgrader is Z, other node is Y)
remove all rabbit/erlang RPM's
Reinstall rabbit software
Update rabbitmqadmin
Enable management plugins (just in case)
Enable auto start.

Rinse and repeat on servers Y, then Z and then start bringing them up
starting with upgrader node. First start Z, then start Y, then start X.
Jason


On Fri, Aug 23, 2013 at 4:51 PM, Jason McIntosh <mcintoshj at gmail.com> wrote:

> Here's some other details from the logs if it might help (note I
> discovered the web-gui issue was a cache issue):
>
> =INFO REPORT==== 23-Aug-2013::15:37:43 ===
> upgrades: Mnesia dir backed up to
> "/data/rabbitmq/rabbitmq/mnesia/cluster-upgrade-backup"
>
> =INFO REPORT==== 23-Aug-2013::15:37:43 ===
> mnesia upgrades: 1 to apply
>
> =INFO REPORT==== 23-Aug-2013::15:37:43 ===
> mnesia upgrades: Breaking cluster
>
> =INFO REPORT==== 23-Aug-2013::15:37:43 ===
> mnesia upgrades: Applying rabbit_upgrade_functions:exchange_decorators
>
> =INFO REPORT==== 23-Aug-2013::15:37:43 ===
> mnesia upgrades: All upgrades applied successfully
>
> =INFO REPORT==== 23-Aug-2013::15:37:44 ===
> Starting RabbitMQ 3.1.5 on Erlang R16B01
> Copyright (C) 2007-2013 GoPivotal, Inc.
> Licensed under the MPL.  See http://www.rabbitmq.com/
>
> =INFO REPORT==== 23-Aug-2013::15:37:44 ===
> node           : cluster at rabbitmqm10t
> home dir       : /var/lib/rabbitmq
> config file(s) : (none)
> cookie hash    : 9fmMn2E9ImSXspwK+tgGgQ==
> log            : /data/rabbitmq/cluster.log
> sasl log       : /data/rabbitmq/cluster-sasl.log
> database dir   : /data/rabbitmq/rabbitmq/mnesia/cluster
>
> =INFO REPORT==== 23-Aug-2013::15:37:45 ===
> Limiting to approx 924 file handles (829 sockets)
>
> =INFO REPORT==== 23-Aug-2013::15:37:45 ===
> upgrades: Mnesia backup removed
>
> =INFO REPORT==== 23-Aug-2013::15:37:45 ===
> Memory limit set to 396MB of 991MB total.
>
> =INFO REPORT==== 23-Aug-2013::15:37:45 ===
> Disk free limit set to 1000MB
>
>
>
> On Fri, Aug 23, 2013 at 4:21 PM, Jason McIntosh <mcintoshj at gmail.com>wrote:
>
>> We just upgraded our Alpha environment to 3.1.5 and Erlang R16B01.  In
>> the process, ALL of our queue definitions disappeared.  Checking logs, I
>> get a  whole lot of errors like the below (names redacted).  Thankfully, I
>> can auto-generate all the stuff again, but I'm having errors on my
>> "upgrader" node right now that is preventing it from even working - the
>> webgui gives a "ReferenceError: queue_length is not defined" error and
>> doesn't display any information. Note, I followed the instructions - picked
>> an upgrader node, shutdown all the other nodes in order, brought the
>> "upgrader" node back online, then the others in sequential order.  This has
>> me really really spooked though.
>>
>> Any advice??
>> Thanks!
>> Jason
>>
>> =ERROR REPORT==== 23-Aug-2013::15:37:50 ===
>> ** Generic server <0.303.0> terminating
>> ** Last message in was {'EXIT',<0.350.0>,normal}
>> ** When Server state == {q,
>>                          {amqqueue,
>>                           {resource,<<"EXCHANGE">>,queue,
>>                            <<"QUEUE">>},
>>                           true,false,none,
>>                           [{<<"x-ha-policy">>,longstr,<<"all">>}],
>>                           <0.303.0>,[],[],
>>                           [{vhost,<<"pca">>},
>>                            {name,<<"HA">>},
>>                            {pattern,<<"^(?!amq\\.).*">>},
>>                            {definition,[{<<"ha-mode">>,<<"all">>}]},
>>                            {priority,0}],
>>                           [{<0.261.0>,<0.260.0>},
>>                            {<0.308.0>,<0.260.0>},
>>                            {<0.299.0>,<0.258.0>},
>>                            {<0.562.0>,<0.561.0>}]},
>>                          none,false,rabbit_mirror_queue_master,
>>                          {state,
>>                           {resource,<<"EXCHANGE">>,queue,
>>                            <<"QUEUE">>},
>>                           <0.351.0>,<0.350.0>,rabbit_variable_queue,
>>                           {vqstate,
>>                            {0,{[],[]}},
>>                            {0,{[],[]}},
>>                            {delta,0,0,0},
>>                            {0,{[],[]}},
>>                            {0,{[],[]}},
>>                            0,
>>                            {0,nil},
>>                            {0,nil},
>>                            {qistate,
>>
>> "/data/rabbitmq/rabbitmq/mnesia/cluster/queues/8FHU53A729K2C3KLMS8XPVKJD",
>>                             {{dict,0,16,16,8,80,48,
>>
>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>>                                []},
>>
>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>>                                 []}}},
>>                              []},
>>                             #Ref<0.0.0.1702>,0,65536,
>>                             #Fun<rabbit_variable_queue.2.81334491>,
>>                             {0,nil}},
>>                            {{client_msstate,msg_store_persistent,
>>
>>  <<247,77,14,2,221,67,18,168,38,83,171,47,207,31,
>>                                187,253>>,
>>                              {dict,0,16,16,8,80,48,
>>
>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>>                                []},
>>
>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>>                                 []}}},
>>                              {state,450623,
>>
>> "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent"},
>>                              rabbit_msg_store_ets_index,
>>
>>  "/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_persistent",
>>                              <0.301.0>,454733,446535,458830,462927},
>>                             {client_msstate,msg_store_transient,
>>
>>  <<12,4,24,179,230,120,71,131,21,190,238,120,9,176,
>>                                231,238>>,
>>                              {dict,0,16,16,8,80,48,
>>
>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>>                                []},
>>
>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>>                                 []}}},
>>                         undefined,undefined,undefined,running}
>> ** Reason for termination ==
>> ** {{badmatch,{error,not_found}},
>>     [{rabbit_mirror_queue_master,stop_all_slaves,2,[]},
>>      {rabbit_mirror_queue_master,delete_and_terminate,2,[]},
>>      {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
>>      {rabbit_amqqueue_process,terminate_shutdown,2,[]},
>>      {gen_server2,terminate,3,[]},
>>      {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
>> ** In 'terminate' callback with reason ==
>> ** normal
>>
>>
>> --
>> Jason McIntosh
>> http://mcintosh.poetshome.com/blog/
>> 573-424-7612
>>
>
>
>
> --
> Jason McIntosh
> http://mcintosh.poetshome.com/blog/
> 573-424-7612
>



-- 
Jason McIntosh
http://mcintosh.poetshome.com/blog/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130823/28d4dd45/attachment.htm>


More information about the rabbitmq-discuss mailing list