[rabbitmq-discuss] Unclean shutdown followed by upgrade causes cluster to no longer come up.

[Adam Pollock]

Hi Simon,

Thanks for your speedy response.

> Do you actually mean rabbitmq.conf? The file is usually called rabbitmq.config. 

Yes, sorry, that was just a typo. It is /etc/rabbitmq/rabbitmq.config.

> Assuming the config file is really /etc/rabbitmq/rabbitmq.config, the only way I can think of for 
> this to happen is if the cluster config file itself is corrupt / empty.

They're definitely not empty, and they don't appear to be corrupt (i.e. they open in vi fine and can 
be read, and deleting them and writing the configuration into a new file doesn't fix the error). 
They have been working previously before the upgrade.

> What does /var/lib/rabbitmq/mnesia/cluster_nodes.config (I think that's the path) contain on the 
> RAM nodes? It should contain the same list of nodes as are in rabbitmq.config. 

Node1 contains:

[root at mq1 ~]# cat /var/lib/rabbitmq/mnesia/node1/cluster_nodes.config
[node1 at mq1,node2 at mq2,node3 at mq3].

However, both node2 and node3 do not have this file present anywhere:

[root at mq3 /var/lib/rabbitmq/mnesia/node3]# ls
msg_store_persistent  msg_store_transient  rabbit_serial  schema_version
[root at mq3 /var/lib/rabbitmq/mnesia/node3]# updatedb
[root at mq3 /var/lib/rabbitmq/mnesia/node3]# locate cluster_nodes.config
[root at mq3 /var/lib/rabbitmq/mnesia/node3]#

There are also node2-upgrade-backup and node3-upgrade-backup directories in 
/var/lib/rabbitmq/mnesia, but these also do not contain and cluster_nodes.config files.

Kind regards,

Adam Pollock.
Lead Software Engineer.

Encoded, Ltd.
T: +44 (0)845 120 9790
F: +44 (0)870 830 1945
E: a.pollock at encoded.co.uk
W: http://www.encoded.co.uk


On 26/03/12 13:52, Simon MacMullen wrote:
> On 26/03/12 13:11, Adam Pollock wrote:
>> Hi,
>
> Hi.
>
>> Node1 started up correctly, but doesn't show any of the
>> other nodes in the cluster status, even though the rabbitmq.conf file
>> has them auto-configured.
>
> This is normal in a cluster upgrade - the upgrade works by breaking the cluster on the first node 
> to start, and then having other nodes rejoin.
>
> Do you actually mean rabbitmq.conf? The file is usually called rabbitmq.config.
>
>> Then, when I try to start up node2 and node3,
>> they both give out the following message in /var/log/rabbit/startup_log:
>>
>> [root at mq2 /etc/rabbitmq]# cat /var/log/rabbitmq/startup_log
>> Activating RabbitMQ plugins ...
>> 6 plugins activated:
>> * amqp_client-2.8.1
>> * mochiweb-1.3-rmq2.8.1-git
>> * rabbitmq_management-2.8.1
>> * rabbitmq_management_agent-2.8.1
>> * rabbitmq_mochiweb-2.8.1
>> * webmachine-1.7.0-rmq2.8.1-hg
>>
>> ****
>> Cluster upgrade needed but this is a ram node.
>> Please first start the last disc node to shut down.
>> ****
>
> So this implies that they think no other cluster nodes are running.
>
> Assuming the config file is really /etc/rabbitmq/rabbitmq.config, the only way I can think of for 
> this to happen is if the cluster config file itself is corrupt / empty. What does 
> /var/lib/rabbitmq/mnesia/cluster_nodes.config (I think that's the path) contain on the RAM nodes? 
> It should contain the same list of nodes as are in rabbitmq.config.
>
> Cheers, Simon
>