[rabbitmq-discuss] Fwd: RabbitMQ 2.8.2 Erlang crash

Wed Jun 13 11:53:10 BST 2012

Hi,

> I have a problem with starting RabbitMQ service after spliting
> cluster.  Software crash everytime during
> "/etc/init.d/rabbitmq-server start".

What do you mean by "splitting cluster"? I'd interpret it as "the
connection was severed between the two nodes", but maybe it's
something different, considering that you're having problems
*starting* nodes. In any case, clustering in rabbit does not tolerate
partitions well. The most it can take is simply nodes going down and
then up again, but if the data in the tables diverge, things will go
wrong.

In your case, since you're getting a timeout in the database boot
step, I'd guess that for some reason mnesia thinks that it has to wait
the tables in the other nodes first. This typically happen when
trying to start a node of an offline cluster before starting the last
node to go down (mnesia wants the last node to go down first because
it has the most up-to-date data). Normally rabbit will terminate with
a friendlier error message ("please start up the last node to go down"
or similar) but if you shut down the nodes abruptly this won't happen
and you'll get an hanging node.

If what I described above matches your situation, then you can refer
to this other post I just sent regarding those situations:
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-June/020594.html
. If it doesn't, can you describe in more detail what you are doing to
get that? The logs would help too.

Francesco.

At Sat, 9 Jun 2012 17:24:47 +0200,
Michał Kiędyś wrote:
> 
> [1  <multipart/alternative (7bit)>]
> [1.1  <text/plain; ISO-8859-2 (quoted-printable)>]
> 
> [1.2  <text/html; ISO-8859-2 (quoted-printable)>]
> 
> ---------- Forwarded message ----------
> From: Alexis Richardson <alexis at rabbitmq.com>
> Date: 2012/6/8
> Subject: Re: RabbitMQ 2.8.2 Erlang crash
> To: Michał Kiędyś / Gadu-Gadu <m.kiedys at gadu-gadu.pl>
> Cc: rabbitmq info <info at rabbitmq.com>
> 
> Michał
> 
> Also, if you could post this to rabbitmq-discuss, that would be helpful.
> 
> Best wishes
> 
> alexis
> 
> On Fri, Jun 8, 2012 at 8:34 AM, Michał Kiędyś / Gadu-Gadu
> <m.kiedys at gadu-gadu.pl> wrote:
> > Hello
> >
> > I have a problem with starting RabbitMQ service after spliting cluster.
> > Software crash everytime during "/etc/init.d/rabbitmq-server start".
> >
> > Log:
> > root at arch-task-mq-8:~# rabbitmq-server
> > Activating RabbitMQ plugins ...
> > 6 plugins activated:
> > * amqp_client-2.8.2
> > * mochiweb-1.3-rmq2.8.2-git
> > * rabbitmq_management-2.8.2
> > * rabbitmq_management_agent-2.8.2
> > * rabbitmq_mochiweb-2.8.2
> > * webmachine-1.7.0-rmq2.8.2-hg
> >
> > ** Found 0 name clashes in code paths
> >
> > +---+   +---+
> > |   |   |   |
> > |   |   |   |
> > |   |   |   |
> > |   +---+   +-------+
> > |                   |
> > | RabbitMQ  +---+   |
> > |           |   |   |
> > |   v2.8.2  +---+   |
> > |                   |
> > +-------------------+
> > AMQP 0-9-1 / 0-9 / 0-8
> > Copyright (C) 2007-2012 VMware, Inc.
> > Licensed under the MPL.  See http://www.rabbitmq.com/
> >
> > node           : rabbit at arch-task-mq-8
> > app descriptor :
> > /usr/lib/rabbitmq/lib/rabbitmq_server-2.8.2/sbin/../ebin/rabbit.app
> > home dir       : /var/lib/rabbitmq
> > config file(s) : /etc/rabbitmq/rabbitmq.config
> > cookie hash    : +MKjvqvf4/0q7+0Fhc9OEw==
> > log            : /var/log/rabbitmq/rabbit at arch-task-mq-8.log
> > sasl log       : /var/log/rabbitmq/rabbit at arch-task-mq-8-sasl.log
> > database dir   : /var/lib/rabbitmq/mnesia/rabbit at arch-task-mq-8
> > erlang version : 5.8
> >
> > -- rabbit boot start
> > starting file handle cache server
> > ...done
> > starting worker pool
> >  ...done
> > starting database                                                     ...
> >
> > BOOT FAILED
> > ===========
> >
> > Timeout contacting cluster nodes: ['rabbit at arch-task-mq-7'].
> >
> > DIAGNOSTICS
> > ===========
> >
> > nodes in question: ['rabbit at arch-task-mq-7']
> >
> > hosts, their running nodes and ports:
> > - arch-task-mq-7: [{rabbit,46044}]
> >
> > current node details:
> > - node name: 'rabbit at arch-task-mq-8'
> > - home dir: /var/lib/rabbitmq
> > - cookie hash: +MKjvqvf4/0q7+0Fhc9OEw==
> >
> >
> > {"Kernel pid
> > terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot}}}}}"}
> >
> > Crash dump was written to: erl_crash.dump
> > Kernel pid terminated (application_controller)
> > ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot}}}}})
> >
> >
> >
> > Regards,
> >
> > --
> > Michał Kiędyś
> > web architect / Pion Aplikacji Internetowych
> >
> > GG Network S.A
> > ul. Kamionkowska 45
> > 03-812 Warszawa
> > tel.: +48 22 427 79 00
> > fax.: +48 22 427 74 25
> > gg:16698
> >
> > Spółka zarejestrowana w Sądzie Rejonowym dla m. st. Warszawy, XIII
> > Wydział Gospodarczy KRS pod numerem 0000264575, NIP 867-19-48-977.
> > Kapitał zakładowy:  1 758 461,10 zł - wpłacony w całości.
> >
> 
> 
> [2  <text/plain; us-ascii (7bit)>]
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss