[rabbitmq-discuss] Unclean shutdown followed by upgrade causes cluster to no longer come up.

[Adam Pollock]

Hi Simon,

Abort abort abort. This was an iptables issue. Your "So this implies that they think no other 
cluster nodes are running" remark gave it away after a few minutes of thinking about it. Someone has 
been on the machine and "hardened" it. Thanks very much for your help though and sorry for taking up 
your time.

Kind regards,

Adam Pollock.
Lead Software Engineer.

Encoded, Ltd.
T: +44 (0)845 120 9790
F: +44 (0)870 830 1945
E: a.pollock at encoded.co.uk
W: http://www.encoded.co.uk


On 26/03/12 14:09, Adam Pollock wrote:
> Hi Simon,
>
> Thanks for your speedy response.
>
>> Do you actually mean rabbitmq.conf? The file is usually called rabbitmq.config. 
>
> Yes, sorry, that was just a typo. It is /etc/rabbitmq/rabbitmq.config.
>
>> Assuming the config file is really /etc/rabbitmq/rabbitmq.config, the only way I can think of for 
>> this to happen is if the cluster config file itself is corrupt / empty.
>
> They're definitely not empty, and they don't appear to be corrupt (i.e. they open in vi fine and 
> can be read, and deleting them and writing the configuration into a new file doesn't fix the 
> error). They have been working previously before the upgrade.
>
>> What does /var/lib/rabbitmq/mnesia/cluster_nodes.config (I think that's the path) contain on the 
>> RAM nodes? It should contain the same list of nodes as are in rabbitmq.config. 
>
> Node1 contains:
>
> [root at mq1 ~]# cat /var/lib/rabbitmq/mnesia/node1/cluster_nodes.config
> [node1 at mq1,node2 at mq2,node3 at mq3].
>
> However, both node2 and node3 do not have this file present anywhere:
>
> [root at mq3 /var/lib/rabbitmq/mnesia/node3]# ls
> msg_store_persistent  msg_store_transient  rabbit_serial  schema_version
> [root at mq3 /var/lib/rabbitmq/mnesia/node3]# updatedb
> [root at mq3 /var/lib/rabbitmq/mnesia/node3]# locate cluster_nodes.config
> [root at mq3 /var/lib/rabbitmq/mnesia/node3]#
>
> There are also node2-upgrade-backup and node3-upgrade-backup directories in 
> /var/lib/rabbitmq/mnesia, but these also do not contain and cluster_nodes.config files.
>
> Kind regards,
>
> Adam Pollock.
> Lead Software Engineer.
>
> Encoded, Ltd.
> T: +44 (0)845 120 9790
> F: +44 (0)870 830 1945
> E: a.pollock at encoded.co.uk
> W: http://www.encoded.co.uk
>
>
> On 26/03/12 13:52, Simon MacMullen wrote:
>> On 26/03/12 13:11, Adam Pollock wrote:
>>> Hi,
>>
>> Hi.
>>
>>> Node1 started up correctly, but doesn't show any of the
>>> other nodes in the cluster status, even though the rabbitmq.conf file
>>> has them auto-configured.
>>
>> This is normal in a cluster upgrade - the upgrade works by breaking the cluster on the first node 
>> to start, and then having other nodes rejoin.
>>
>> Do you actually mean rabbitmq.conf? The file is usually called rabbitmq.config.
>>
>>> Then, when I try to start up node2 and node3,
>>> they both give out the following message in /var/log/rabbit/startup_log:
>>>
>>> [root at mq2 /etc/rabbitmq]# cat /var/log/rabbitmq/startup_log
>>> Activating RabbitMQ plugins ...
>>> 6 plugins activated:
>>> * amqp_client-2.8.1
>>> * mochiweb-1.3-rmq2.8.1-git
>>> * rabbitmq_management-2.8.1
>>> * rabbitmq_management_agent-2.8.1
>>> * rabbitmq_mochiweb-2.8.1
>>> * webmachine-1.7.0-rmq2.8.1-hg
>>>
>>> ****
>>> Cluster upgrade needed but this is a ram node.
>>> Please first start the last disc node to shut down.
>>> ****
>>
>> So this implies that they think no other cluster nodes are running.
>>
>> Assuming the config file is really /etc/rabbitmq/rabbitmq.config, the only way I can think of for 
>> this to happen is if the cluster config file itself is corrupt / empty. What does 
>> /var/lib/rabbitmq/mnesia/cluster_nodes.config (I think that's the path) contain on the RAM nodes? 
>> It should contain the same list of nodes as are in rabbitmq.config.
>>
>> Cheers, Simon
>>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>