[rabbitmq-discuss] Recurring partitioning problem on local network

[Bill Chmura]

One of our sets went down today

Both nodes basically have this, just naming the other node:

=INFO REPORT==== 10-Dec-2013::18:29:24 ===
rabbit on node 'rabbit at NURWEB-QAWEB01' down

=ERROR REPORT==== 10-Dec-2013::18:29:35 ===
Mnesia('rabbit at NURWEB-QAWEB02'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit at NURWEB-QAWEB01'}

=INFO REPORT==== 10-Dec-2013::18:29:47 ===
node 'rabbit at NURWEB-QAWEB01' down: connection_closed

Not much more info with the patched base file... does this help at all?

I tried searching and got a lot on connection closed abruptly... but it did not sound right.



-----Original Message-----
From: Simon MacMullen [mailto:simon at rabbitmq.com] 
Sent: Tuesday, December 10, 2013 12:09 PM
To: Bill Chmura
Cc: Discussions about RabbitMQ
Subject: Re: [rabbitmq-discuss] Recurring partitioning problem on local network

On 10/12/13 16:56, Bill Chmura wrote:
> What do you mean by a patched version?  Are there instructions?

I've attached a modified rabbit_node_monitor.beam which can replace the one in your RabbitMQ installation's ebin directory. This version has been modified to additionally log messages on node down like this:

> =INFO REPORT==== 10-Dec-2013::16:32:02 === node hare at smacmullen down: 
> connection_closed

giving the reason why the runtime considers the other node to be down. 
(This feature will make its way into a future release eventually). I am curious as to whether you are really seeing net_ticktime-related timeouts or something else, and it would be good to get some data on that.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal

This email and all attachments are confidential and intended solely 
for the use of the individual or entity to which they are addressed. 
If you have received this email in error please notify the sender 
by replying to this message. If you are not the intended recipient, 
please delete this message and all attachments immediately.  Do not 
copy, disclose, use or act upon the information contained. Please 
note that any views or opinions presented in this email are solely 
those of the author and do not necessarily represent those of the 
company. Finally, the recipient should check this email and any 
attachments for the presence of viruses. While every attempt is made 
to verify that the contents are safe, the company accepts no liability 
for any damage caused by any virus transmitted by this email.