<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Thanks for reply.<br>
<i><br>
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource
temporarily
<br>
unavailable
<br>
</i><br>
</tt><tt>I've checked that no limit of processes was reached (over
300 proc. were running),<br>
I was able to start new processes. May be there was a file lock on
one of the (configuration?) files?</tt><br>
<tt>This error disappeared when local rabbitmq broker have become
able to see other brokers.<br>
<br>
Most of all I'm interested why 3 nodes crashed at once and why
following message appeared:<br>
</tt><tt> <br>
=ERROR REPORT==== 14-Feb-2012::19:13:47 ===<br>
** Generic server <0.1093.0> terminating<br>
** Last message in was {mnesia_locker,wosnfs@dell39,granted}<br>
** When Server state == 5<br>
** Reason for termination == <br>
** {unexpected_info,{mnesia_locker,wosnfs@dell39,granted}}<br>
<br>
Following error messages may be the consuquece of this.</tt><br>
<tt>Look forward to results of your investigation</tt>.<br>
<br>
<tt>P.S. better way to recover cluster is (to leave cluster if
possible), clear mnesia on crashed nodes and rejoin them?</tt><br>
<br>
On 02/22/2012 08:07 PM, Simon MacMullen wrote:
<blockquote cite="mid:4F4520DD.6040302@rabbitmq.com" type="cite">On
22/02/12 16:13, Artsiom wrote:
<br>
<blockquote type="cite">/usr/lib/rabbitmq/bin/rabbitmq-env: fork:
retry: Resource temporarily
<br>
unavailable
<br>
</blockquote>
<br>
This implies that you're running into some OS-imposed limit on the
number of (OS not Erlang) processes. When you do that, Bad Things
are likely to happen.
<br>
<br>
I also see a bunch of errors along the lines of:
<br>
<br>
=ERROR REPORT==== 14-Feb-2012::20:51:30 ===
<br>
Mnesia(wosnfs@dell39): ** ERROR ** mnesia_event got
{inconsistent_database, running_partitioned_network,
wosnfs@dell33}
<br>
<br>
in machine 4's log, which is another bad sign - RabbitMQ clusters
are not designed to tolerate network partitions.
<br>
<br>
I'll look into some of the other error messages that are showing
up in your logs; it's possible some of then are real, but you have
some bad things going on underneath RabbitMQ - you need to sort
those out first,
<br>
<br>
Cheers, Simon
<br>
<br>
</blockquote>
<tt>--<br>
Best regards,<br>
Artsiom</tt><br>
</body>
</html>