[rabbitmq-discuss] can't restart rabbit cluster after power outage

[Simon MacMullen]

On 20/06/2014 19:00, Ben Hsu wrote:
> That's interesting. It looks like an older version of rabbitmq
>        {rabbit,"RabbitMQ","3.0.0"},

> Now thats interesting! Having an erlang node up without having to start
> up rabbit sounds very useful. I tried doing forgetting the cluster node
> and got this:
>
> sudo rabbitmqctl forget_cluster_node --offline rabbit at node3
>
> Error: not_last_node_to_go_down: The node you are trying to remove from
> was not the last to go down (excluding the node you are removing).
> Please use the the last node to go down to remove nodes when the cluster
> is offline.

Ah. Yes, that used to be a constraint for forget_cluster_node; it was 
removed in 3.1.4.

> I got the same error when I tried the same command with each of the
> other nodes. So it looks like rabbit has no idea which node was the last
> to die. Is it possible to ask rabbit for this information, possibly by
> groveling around mnesia with "rabbitmqctl eval"?

I think on 3.0.0 you might be able to get away with:

1) Start with NODE_ONLY
2) rabbitmqctl eval 'application:start(mnesia).'
3) rabbitmqctl eval 'rabbit_table:force_load().'
4) rabbitmqctl eval 'rabbit_table:wait_for_replicated().'
5) rabbitmqctl eval 'application:stop(mnesia).'

Then stop, remove NODE_ONLY and start again. That won't have removed 
node3 from the cluster so you'll need to do that with 
forget_cluster_node separately afterwards - but you can do that online 
once node1 and node2 are running.

> I'm also okay with a drastic solution like "blow away the mnesia
> database and recover the data in some other way". Part of the reason I'm
> asking all these questions is because I want to learn more about rabbitmq

There's no real way to recover from deleting the database other than to 
start from scratch.

Cheers, Simon