[rabbitmq-discuss] Outage, brokers refuse to start back up

Fri Dec 9 15:03:56 GMT 2011

Okay, so I backed up our mnesia dir and wiped it clean on both boxes.
Brokers started fine. I then copied the DCD and DCL files back over
and my users, exchanges, and queues were back.

Is there a recommended way to protect against such an outage?

Thanks,
James

On Fri, Dec 9, 2011 at 8:41 AM, James Carr <james.r.carr at gmail.com> wrote:
> So our datacenter had a power failure yesterday and the brokers now
> REFUSE to start back up. I can send the erl_crash.dump as needed (it
> is quite large) but here are the other logs.
>
> Any ideas?
>
> Thanks,
> James
>
>
> Rabbitmq log:
>
> =INFO REPORT==== 9-Dec-2011::08:35:32 ===
> Limiting to approx 924 file handles (829 sockets)
>
> =ERROR REPORT==== 9-Dec-2011::08:36:02 ===
> FAILED
> Reason: {error,
>            {timeout_waiting_for_tables,
>                [rabbit_user,rabbit_user_permission,rabbit_vhost,
>                 rabbit_durable_route,rabbit_durable_exchange,
>                 rabbit_durable_queue]}}
> Stacktrace: [{rabbit_mnesia,wait_for_tables,1},
>             {rabbit_mnesia,check_schema_integrity,0},
>             {rabbit_mnesia,ensure_schema_integrity,0},
>             {rabbit_mnesia,init,0},
>             {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
>             {rabbit,run_boot_step,1},
>             {rabbit,'-start/2-lc$^0/1-0-',1},
>             {rabbit,start,2}]
>
> =INFO REPORT==== 9-Dec-2011::08:36:03 ===
>    application: rabbit
>    exited: {bad_return,{{rabbit,start,[normal,[]]},
>                         {'EXIT',{rabbit,failure_during_boot}}}}
>    type: permanent
>
>
> startup_log:
>
> Activating RabbitMQ plugins ...
> 11 plugins activated:
> * amqp_client-2.6.1
> * erlando-2.6.1
> * mochiweb-1.3-rmq2.6.1-git9a53dbd
> * rabbitmq_federation-2.6.1
> * rabbitmq_management-2.6.1
> * rabbitmq_management_agent-2.6.1
> * rabbitmq_management_visualiser-2.6.1
> * rabbitmq_mochiweb-2.6.1
> * rabbitmq_shovel-2.6.1
> * rabbitmq_shovel_management-2.6.1
> * webmachine-1.7.0-rmq2.6.1-hg0c4b60a
>
>
> +---+   +---+
> |   |   |   |
> |   |   |   |
> |   |   |   |
> |   +---+   +-------+
> |                   |
> | RabbitMQ  +---+   |
> |           |   |   |
> |   v2.6.1  +---+   |
> |                   |
> +-------------------+
> AMQP 0-9-1 / 0-9 / 0-8
> Copyright (C) 2007-2011 VMware, Inc.
> Licensed under the MPL.  See http://www.rabbitmq.com/
>
> node           : rabbit at brokerm02p
> app descriptor :
> /usr/lib/rabbitmq/lib/rabbitmq_server-2.6.1/sbin/../ebin/rabbit.app
> home dir       : /var/lib/rabbitmq
> config file(s) : /etc/rabbitmq/rabbitmq.config
> cookie hash    : Mg6GXWPn9Lrj9HC/D14CWA==
> log            : /var/log/rabbitmq/rabbit at brokerm02p.log
> sasl log       : /var/log/rabbitmq/rabbit at brokerm02p-sasl.log
> database dir   : /var/lib/rabbitmq/mnesia/rabbit at brokerm02p
> erlang version : 5.8.4
>
> -- rabbit boot start
> starting file handle cache server                                     ...done
> starting worker pool                                                  ...done
> starting database
> ...BOOT ERROR: FAILED
> Reason: {error,
>            {timeout_waiting_for_tables,
>                [rabbit_user,rabbit_user_permission,rabbit_vhost,
>                 rabbit_durable_route,rabbit_durable_exchange,
>                 rabbit_durable_queue]}}
> Stacktrace: [{rabbit_mnesia,wait_for_tables,1},
>             {rabbit_mnesia,check_schema_integrity,0},
>             {rabbit_mnesia,ensure_schema_integrity,0},
>             {rabbit_mnesia,init,0},
>             {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
>             {rabbit,run_boot_step,1},
>             {rabbit,'-start/2-lc$^0/1-0-',1},
>             {rabbit,start,2}]
> {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot}}}}}"}
>
> startup_err:
>
> Erlang has closed
>
> Crash dump was written to: erl_crash.dump
> Kernel pid terminated (application_controller)
> ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot}}}}})