<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hello everybody.<br>
<br>
I have RabbitMQ cluster across 4 machines. Then 2 machines were
added to cluster.<br>
Everything was ok, but after few hours of load three of them
crashed almost simulteneously (according to log).<br>
Besides at crash time memory consumption increased 3 times.<br>
<br>
Log files from crashed machines:<br>
<br>
(dell33) - machine 1<br>
</tt>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="http://pastebin.com/A1rdBSkN">http://pastebin.com/A1rdBSkN</a><br>
<br>
<tt># rabbitmqctl cluster_status<br>
Error: unable to connect to node wosnfs@dell33: nodedown<br>
diagnostics:<br>
- nodes and their ports on dell33: [{rabbitmqctl15567,47802},<br>
{rabbitmqctl15576,41488},<br>
{rabbitmqctl16776,44227},<br>
{rabbitmqctl16634,41524},<br>
{rabbitmqctl20944,36980},<br>
{rabbitmqctl20942,41667},<br>
{rabbitmqctl8262,44050}]<br>
- current node: rabbitmqctl8262@dell33<br>
- current node home dir: /var/lib/rabbitmq<br>
- current node cookie hash: FwHV/NurYGNpetHE+jQlMQ==</tt><br>
<br>
<tt>(dell34) - machine 2</tt><br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="http://pastebin.com/36ihKNqv">http://pastebin.com/36ihKNqv</a><br>
<br>
<tt># rabbitmqctl list_queues<br>
<br>
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource
temporarily unavailable<br>
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource
temporarily unavailable<br>
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource
temporarily unavailable<br>
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource
temporarily unavailable</tt><br>
<br>
<tt>(dell38) - machine 3</tt><br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="http://pastebin.com/TfWBcSus">http://pastebin.com/TfWBcSus</a><br>
<br>
<tt>Log file from one of survived machines:</tt><br>
<br>
<tt>(dell39) - machine 4</tt><br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="http://pastebin.com/ij1k4L9K">http://pastebin.com/ij1k4L9K</a><br>
<br>
<tt>Above mentioned issue appeared on rabbitmq 2.6.1, but sometimes
it appears on 2.7.1.<br>
Queues, which names are in log (queue-batch-{3,4,5,6}), are
durable with "x-ha-policy" property.<br>
All rabbitmq brokers are clustered as disk nodes.</tt><br>
<br>
<tt>Could anybody help me to examine log files and find out the
reason?</tt><br>
<br>
<tt>--<br>
Best regards,<br>
Artsiom</tt><br>
</body>
</html>