<div dir="ltr"><div>Hi there,</div><div><br></div><div>Hoping someone can help me out. We recently experienced 2 crashes with our RabbitMQ cluster. After the first crash, we moved the Mnesia directories elsewhere, and started RabbitMQ again. This got us up and running. Second time it happened, we had the original nodes plus an additional 5 nodes we had added to the cluster that we were planning to leave in place while shutting the old nodes down.</div>
<div><br></div><div>During the crash symptoms were as follows:</div><div><br></div><div>- Escalating (and sudden) CPU utilisation on some (but not all) nodes</div><div>- Escalation memory usage (not necessarily aligned to the spiking CPU)</div>
<div>- Increasing time to publish on queues (and specifically on a test queue we have setup that exists only to test publishing and consuming from the cluster hosts)</div><div>- Running `rabbitmqctl cluster status` gets increasingly slow (some nodes eventually taking up to 10m to return with the response data - some were fast and took 5s)</div>
<div>- Management plugin stops responding / or responding so slowly it's no longer loading any data at all (probably same thing that causes the preceeding item)</div><div>- Can't force nodes to forget other nodes (calling `rabbitmqctl forget_cluster_node` doesn't return)</div>
<div><br></div><div>- When trying to shut down a node, running `rabbitmqctl stop_app` appears to block on epmd and doesn't return</div><div>--- When that doesn't return we eventually have to ctrl-c the command</div>
<div>--- We have to issue a kill signal to rabbit to stop it</div><div>--- Do the same to the epmd process</div><div>--- However the other nodes all still think that the killed node is active (based on `rabbitmqctl cluster status` -- both nodes slow to run this, and fast to run it saw the same view of the cluster that included the dead node)</div>
<div><br></div><div><br></div><div>Config / details as follows (we use mirrored queues -- 5 hosts, all disc nodes, with a global policy that all queues are mirrored "ha-mode:all"), running on Linux</div><div><br>
</div><div>[</div><div> {rabbit, [</div><div> {cluster_nodes, {['rabbit@b05.internal', 'rabbit@b06.internal','rabbit@b07.internal','rabbit@b08.internal','rabbit@b09.internal'], disc}},</div>
<div> {cluster_partition_handling, pause_minority}</div><div> ]}</div><div>]</div><div><br></div><div>And the env:</div><div><br></div><div>NODENAME="rabbit@b09.internal"</div><div>SERVER_ERL_ARGS="-kernel inet_dist_listen_min 27248 -kernel inet_dist_listen_max 27248"</div>
<div><br></div><div>The erl_crash.dump slogan error was : eheap_alloc: Cannot allocate 2850821240 bytes of memory (of type "old_heap").</div><div><br></div><div>System version : Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]</div>
<div><br></div><div>Compiled<span style="white-space:pre-wrap"> </span>Fri Dec 16 03:22:15 2011</div><div>Taints<span style="white-space:pre-wrap"> </span>(none)</div><div>Memory allocated<span style="white-space:pre-wrap"> </span>6821368760 bytes</div>
<div>Atoms<span style="white-space:pre-wrap"> </span>22440</div><div>Processes<span style="white-space:pre-wrap"> </span>4899</div><div>ETS tables<span style="white-space:pre-wrap"> </span>80</div><div>Timers<span style="white-space:pre-wrap"> </span>23</div>
<div>Funs<span style="white-space:pre-wrap"> </span>3994</div><div><br></div><div>When I look at the Process Information it seems there's a small number with ALOT of messages queued, and the rest are an order of magnitude lower:</div>
<div><br></div><div>Pid<span style="white-space:pre-wrap"> </span>Name/Spawned as<span style="white-space:pre-wrap"> </span>State<span style="white-space:pre-wrap"> </span>Reductions<span style="white-space:pre-wrap"> </span>Stack+heap<span style="white-space:pre-wrap"> </span> MsgQ Length</div>
<div><0.400.0><span style="white-space:pre-wrap"> </span>proc_lib:init_p/5<span style="white-space:pre-wrap"> </span>Scheduled<span style="white-space:pre-wrap"> </span>146860259<span style="white-space:pre-wrap"> </span> 59786060<span style="white-space:pre-wrap"> </span>37148</div>
<div><0.373.0><span style="white-space:pre-wrap"> </span>proc_lib:init_p/5<span style="white-space:pre-wrap"> </span>Scheduled<span style="white-space:pre-wrap"> </span>734287949<span style="white-space:pre-wrap"> </span> 1346269<span style="white-space:pre-wrap"> </span>23360</div>
<div><0.366.0><span style="white-space:pre-wrap"> </span>proc_lib:init_p/5<span style="white-space:pre-wrap"> </span>Waiting<span style="white-space:pre-wrap"> </span>114695635<span style="white-space:pre-wrap"> </span> 5135590<span style="white-space:pre-wrap"> </span>19744</div>
<div><0.444.0><span style="white-space:pre-wrap"> </span>proc_lib:init_p/5<span style="white-space:pre-wrap"> </span>Waiting<span style="white-space:pre-wrap"> </span>154538610<span style="white-space:pre-wrap"> </span> 832040<span style="white-space:pre-wrap"> </span> 3326</div>
<div><br></div><div>when I view the second process (first one crashes erlang on me), I see a large number of sender_death events (not sure if these are common or highly unusual ?)</div><div><br></div><div>{'$gen_cast',{gm,{sender_death,<2710.20649.64>}}}</div>
<div><br></div><div>mixed in with other more regular events:</div><div><br></div><div>{'$gen_cast',</div><div> {gm,{publish,<2708.20321.59>,</div><div> {message_properties,undefined,false},</div>
<div> {basic_message,</div><div><.. snip..></div></div>