[rabbitmq-discuss] Complete cluster crash (2.7.1)

[Christian Bick]

Hi,

we had a complete crash of our 3-Node 2.7.1 cluster some days ago. What I saw on the web, this issue might be related to http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-April/019765.html . As such it may have been fixed in 2.8.x which we are going to migrate to. Nevertheless, I would like to ask for clarification on that if possible.

Detailed description:

OS: Linux
Erlang Version: R14A
Rabbit Version: 2.7.1
Cluster Setup: 3 Nodes, all disc mode
Queues: HA mode “all” and durable

The cluster used to be stable for some months with relatively low usage.

Some days before the crash we were doing some OS related maintenance on the server. So we shut down, updated and reset each node one by one before rejoining it to the cluster. After that, some queues were not in synch any more with all other nodes (some synched with one, some synched with both other nodes). We regarded this as none-critical state and ignored it with respect to our already planned migration to 2.8.2.

You can find the relevant logs and the crash dump of Node-1 in the attachments.

Thank you already in advance for your help.

Greetings,
Christian Bick
-------------------------------------------------------------------------------
xing: https://www.xing.com/profile/Christian_Bick2
-------------------------------------------------------------------------------
ZANOX.de AG | Stralauer Allee 2 | 10245 Berlin
www.zanox.com<http://www.zanox.com/> | blog.zanox.com

[disclaimer logo: ZANOX.de AG]

STRAIGHT TO PERFORMANCE
--------------------------------------------------------------------------------

ZANOX.de AG | Headquarters: Berlin AG Charlottenburg | HRB 75459 | VAT identification number: DE 209981705
Executive Board: Philipp Justus (CEO) | Daniel Keller (CTO) | Christian Kleinsorge (CSO) | Joachim Piroth (CFO)
Chairman of the Supervisory Board: Ralph Büchi

This e-mail and any attachments may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail from your system.
Any other use, copying, disclosure or distribution is strictly forbidden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120521/5d81eba8/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node1.log
Type: application/octet-stream
Size: 109252 bytes
Desc: node1.log
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120521/5d81eba8/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node2.log
Type: application/octet-stream
Size: 15437 bytes
Desc: node2.log
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120521/5d81eba8/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node3.log
Type: application/octet-stream
Size: 11057 bytes
Desc: node3.log
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120521/5d81eba8/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: erl_crash.dump.gz
Type: application/x-gzip
Size: 173780 bytes
Desc: erl_crash.dump.gz
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120521/5d81eba8/attachment.bin>