[rabbitmq-discuss] Cluster upgrade failed

Thu May 9 15:52:35 BST 2013

Hi Simon,
Here's what happened... I followed the cluster upgrade instructions by
taking down each server (service rabbitmg-server stop) in order 1,2,3,4.
Upgraded and restarted in reverse order 4,3,2,1.  When they came back up
I could only login as guest.  All users and vhosts are gone, 2 and 3
were clustered and 1 and 4 were standalone. They are behind an HAProxy
so when I would refresh the management UI I see only nodes 2 and 3
clustered, then only 1, then only 4, etc...

I recreated the cluster, join_cluster rabbit01 -> rabbit02 then rabbit04
-> rabbit02.  The cluster looked good but still no users or fabric.  It
did however, show that one of the nodes was using 16GB of disk and I
knew that was persistent messages that were in a queue when I took
everything down.  I started poking around on all the servers and all the
config and db files seemed to be in place .  I was just about to reset
everything when I found the problem....

The upgrade apparently created a new rabbitmq-env file and a new
symbolic link /usr/lib/rabbitmq/bin/rabbitmq-env ->
../lib/rabbitmq_server-3.1.0/sbin/rabbitmq-env.  Our 3.0.4 configuration
had the following entries in rabbitmq-env:

RABBITMQ_MNESIA_BASE=/mnt/data/rabbitmq/mnesia
export RABBITMQ_MNESIA_BASE

Those two entries were not in the new rabbitmq-env file so when rabbit
started after the upgrade the mnesia database was reinitialized in the
default location.  I took down all the servers, edited the file on each
server, brought them back up and then I had a big phat rabbit, just the
way I like it.

Tracy

-----Original Message-----
From: Simon MacMullen [mailto:simon at rabbitmq.com] 
Sent: Thursday, May 09, 2013 9:42 AM
To: Discussions about RabbitMQ
Cc: Tracy Dalzell; Daniel Buchko
Subject: Re: [rabbitmq-discuss] Cluster upgrade failed

Hi. So this looks very much like each node was somehow reset during the
upgrade process.

What exactly did you do for each upgrade?

Cheers, Simon

On 08/05/13 18:13, Tracy Dalzell wrote:
> Status reports attached
>
> *From:*Daniel Buchko [mailto:dbuchko at gopivotal.com]
> *Sent:* Wednesday, May 08, 2013 12:48 PM
> *To:* rabbitmq-discuss at lists.rabbitmq.com
> *Cc:* Tracy Dalzell
> *Subject:* Fwd: Cluster upgrade failed
>
> Forwarding on behalf of Tracy....
>
> ---------- Forwarded message ----------
> From: *Tracy Dalzell* <tracy at guisolutions.com 
> <mailto:tracy at guisolutions.com>>
> Date: Wed, May 8, 2013 at 11:59 AM
> Subject: Cluster upgrade failed
> To: rabbitmq-discuss at lists.rabbitmq.com
> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>
> Hello,
>
> I just upgraded a four node cluster from 3.0.4 -> 3.1.  I took them 
> down (stopped the rabbit server) in order 1,2,3,4.  Upgraded and 
> restarted in reverse order 4,3,2,1.  When everything came back up I 
> could only login as guest.  All users and vhosts are gone.  When I 
> refresh the management UI I see only nodes 2 and 3, then only 1, then 
> only 4, etc...
>
> TIA,
>
> Tracy
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

--
Simon MacMullen
RabbitMQ, Pivotal