[rabbitmq-discuss] RabbitMQ node stopped while broker still up

[Wong Kam Hoong]

Hi Simon,

I checked the log and noticed every time when start this particular
RabbitMQ, the log showed some error messages as follow:

=ERROR REPORT==== 4-Oct-2012::05:46:57 ===
Mnesia(nplay at app2): ** ERROR ** mnesia_event got {inconsistent_database,
starting_partitioned_network, nplay at web1}

=ERROR REPORT==== 4-Oct-2012::05:46:57 ===
Mnesia(nplay at app2): ** ERROR ** mnesia_event got {inconsistent_database,
starting_partitioned_network, nplay at web2}

Why the errors logged? Is it related to my cluster setting?

This is production environment, I can guaranteed no one dare to execute the
command "rabbitmqctl stop".

Regards,
Wong

On Tue, Oct 16, 2012 at 6:33 PM, Simon MacMullen <simon at rabbitmq.com> wrote:

> Hmm. The SASL log will not necessarily have to contain anything, it is
> really more of an error log. So I guess there was no error.
>
> Aha! The log contains:
>
>   =INFO REPORT==== 16-Oct-2012::00:48:36 ===
>   Halting Erlang VM
>
> We only log that after invocation of "rabbitmqctl stop". So the reason
> that node shut down was, umm, someone told it to.
>
> And regarding network partitions, we get information about that from
> Mnesia. Mnesia will log something like:
>
>   =ERROR REPORT==== 16-Oct-2012::00:04:19 ===
>   Mnesia(nplay at app2): ** ERROR ** mnesia_event got
>     {inconsistent_database, running_partitioned_network, nplay at web2}
>
> when it has detected a network partition. Note the
> "running_partitioned_network" - it will also log a very similar message
> with "starting_partitioned_network" the first time it starts *after* a
> partition.
>
> Future versions of RabbitMQ will make this information more accessible.
>
> Cheers, Simon
>
>
> On 16/10/12 10:32, Wong Kam Hoong wrote:
>
>> Hi Simon,
>>
>> I check in the sasl log, but it not being update for long time, the
>> issue happened at 16-oct-2012 but the latest info showed in the log is
>> only up to 2-oct-2012.
>>
>> Attached is the requested sasl log for the server.
>>
>> Yea, I remember you mentioned before RabbitMQ not recommend to run in
>> partitioned network, we still waiting network team to tell us whether
>> those RabbitMQs is it really deployed in a partitioned network..
>>
>> Just curios, how RabbitMQ identify whether the nodes deployed
>> in partitioned network? I asked this question so that I can discuss
>> better with network team.
>>
>> Regards,
>> Wong
>>
>>
>> On Tue, Oct 16, 2012 at 5:10 PM, Simon MacMullen <simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>> wrote:
>>
>>     Hi. There's nothing in that log to indicate why the node shut down -
>>     can you post the sasl log somewhere?
>>
>>     I don't know if it's related to the network partition. But please
>>     bear in mind that network partitions are really bad for RabbitMQ
>>     clusters.
>>
>>     Cheers, Simon
>>
>>
>>     On 16/10/12 03:20, Wong Kam Hoong wrote:
>>
>>         Hi RabbitMQ Team,
>>
>>         This morning while I checked the RabbitMQs status through web
>>         admin, I
>>         found that one of the RabbitMQ node stopped.
>>
>>         RabbitMQ v2.8.7
>>         Erlang v*R14B04*
>>         *Cluster: Yes, 3 RabbitMQs*
>>
>>
>>         Attached is the log for your reference.
>>
>>         After I restarted the service, then everything back to normal.
>>
>>         I wonder is the problem related to partitioned network:
>>
>>         http://rabbitmq.1065348.n5.__n**abble.com/Statistics-__**
>> database-could-not-be-__**contacted-Message-rates-and-__**
>> queue-lengths-will-not-be-__**shown-td22331.html<http://nabble.com/Statistics-__database-could-not-be-__contacted-Message-rates-and-__queue-lengths-will-not-be-__shown-td22331.html>
>>
>>         <http://rabbitmq.1065348.n5.**nabble.com/Statistics-**
>> database-could-not-be-**contacted-Message-rates-and-**
>> queue-lengths-will-not-be-**shown-td22331.html<http://rabbitmq.1065348.n5.nabble.com/Statistics-database-could-not-be-contacted-Message-rates-and-queue-lengths-will-not-be-shown-td22331.html>
>> >
>>
>>         Thanks & Regards,
>>         Wong
>>
>>
>>         ______________________________**___________________
>>         rabbitmq-discuss mailing list
>>         rabbitmq-discuss at lists.__rabbi**tmq.com <http://rabbitmq.com>
>>         <mailto:rabbitmq-discuss@**lists.rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
>> >
>>         https://lists.rabbitmq.com/__**cgi-bin/mailman/listinfo/__**
>> rabbitmq-discuss<https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss>
>>
>>         <https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**
>> rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>> >
>>
>>
>>
>>     --
>>     Simon MacMullen
>>     RabbitMQ, VMware
>>
>>
>>
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121017/06d3e598/attachment.htm>