[rabbitmq-discuss] Cluster upgrade failed
tracy at guisolutions.com
Thu May 9 15:52:35 BST 2013
Here's what happened... I followed the cluster upgrade instructions by
taking down each server (service rabbitmg-server stop) in order 1,2,3,4.
Upgraded and restarted in reverse order 4,3,2,1. When they came back up
I could only login as guest. All users and vhosts are gone, 2 and 3
were clustered and 1 and 4 were standalone. They are behind an HAProxy
so when I would refresh the management UI I see only nodes 2 and 3
clustered, then only 1, then only 4, etc...
I recreated the cluster, join_cluster rabbit01 -> rabbit02 then rabbit04
-> rabbit02. The cluster looked good but still no users or fabric. It
did however, show that one of the nodes was using 16GB of disk and I
knew that was persistent messages that were in a queue when I took
everything down. I started poking around on all the servers and all the
config and db files seemed to be in place . I was just about to reset
everything when I found the problem....
The upgrade apparently created a new rabbitmq-env file and a new
symbolic link /usr/lib/rabbitmq/bin/rabbitmq-env ->
../lib/rabbitmq_server-3.1.0/sbin/rabbitmq-env. Our 3.0.4 configuration
had the following entries in rabbitmq-env:
Those two entries were not in the new rabbitmq-env file so when rabbit
started after the upgrade the mnesia database was reinitialized in the
default location. I took down all the servers, edited the file on each
server, brought them back up and then I had a big phat rabbit, just the
way I like it.
From: Simon MacMullen [mailto:simon at rabbitmq.com]
Sent: Thursday, May 09, 2013 9:42 AM
To: Discussions about RabbitMQ
Cc: Tracy Dalzell; Daniel Buchko
Subject: Re: [rabbitmq-discuss] Cluster upgrade failed
Hi. So this looks very much like each node was somehow reset during the
What exactly did you do for each upgrade?
On 08/05/13 18:13, Tracy Dalzell wrote:
> Status reports attached
> *From:*Daniel Buchko [mailto:dbuchko at gopivotal.com]
> *Sent:* Wednesday, May 08, 2013 12:48 PM
> *To:* rabbitmq-discuss at lists.rabbitmq.com
> *Cc:* Tracy Dalzell
> *Subject:* Fwd: Cluster upgrade failed
> Forwarding on behalf of Tracy....
> ---------- Forwarded message ----------
> From: *Tracy Dalzell* <tracy at guisolutions.com
> <mailto:tracy at guisolutions.com>>
> Date: Wed, May 8, 2013 at 11:59 AM
> Subject: Cluster upgrade failed
> To: rabbitmq-discuss at lists.rabbitmq.com
> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
> I just upgraded a four node cluster from 3.0.4 -> 3.1. I took them
> down (stopped the rabbit server) in order 1,2,3,4. Upgraded and
> restarted in reverse order 4,3,2,1. When everything came back up I
> could only login as guest. All users and vhosts are gone. When I
> refresh the management UI I see only nodes 2 and 3, then only 1, then
> only 4, etc...
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
More information about the rabbitmq-discuss