[rabbitmq-discuss] RabbitMQ cluster can't startup

[Doug]

Hello,

We have 3 nodes as a cluster.
But today due to a power off all 3 hosts are down.
After the hosts get up we startup the rabbitmq on each server.
But they can't get started. the startup_log says as below.
Can you help? Thanks.


node           : rabbit at default74
app descriptor :
/usr/lib/rabbitmq/lib/rabbitmq_server-3.0.1/sbin/../ebin/rabbit.app
home dir       : /var/lib/rabbitmq
config file(s) : (none)
cookie hash    : gK39sZ9ArVpWGzGJIi4ZrQ==
log            : /var/log/rabbitmq/rabbit at default74.log
sasl log       : /var/log/rabbitmq/rabbit at default74-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit at default74
erlang version : 5.9.1

-- rabbit boot start
starting file handle cache server                                     ...done
starting worker pool                                                  ...done
starting database                                                     ...

BOOT FAILED
===========

Timeout contacting cluster nodes: [rabbit at default73,rabbit at default72].

DIAGNOSTICS
===========

nodes in question: [rabbit at default73,rabbit at default72]

hosts, their running nodes and ports:
- default72: [{ns_1,21100}]
- default73: [{ns_1,21100}]

current node details:
- node name: rabbit at default74
- home dir: /var/lib/rabbitmq
- cookie hash: gK39sZ9ArVpWGzGJIi4ZrQ==




BOOT FAILED
===========

Error description:
   {could_not_start,rabbit,
       {bad_return,
           {{rabbit,start,[normal,[]]},
            {'EXIT',
                {rabbit,failure_during_boot,
                    {error,
                        {timeout_waiting_for_tables,
                            [rabbit_user,rabbit_user_permission,rabbit_vhost,
                             rabbit_durable_route,rabbit_durable_exchange,
                             rabbit_runtime_parameters,
                             rabbit_durable_queue]}}}}}}}

Log files (may contain more information):
   /var/log/rabbitmq/rabbit at default74.log
   /var/log/rabbitmq/rabbit at default74-sasl.log

{"init terminating in
do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}}}}}}}}}