[rabbitmq-discuss] RabbitMQ crashed again

[Matthew Sackman]

On Wed, Mar 31, 2010 at 05:37:40PM -0500, Ian Ragsdale wrote:
> Thanks for confirming, Michael.  Is it safe to assume that closing a connection will deallocate any associated resources?  I have the example for how to increase the number of processes, so I can increase that and make sure that the connections get reopened on a regular basis, which should hopefully keep whatever is causing this slow resource from crashing the server.

Your reference to system_limits makes me wonder if you've run out of
ets tables. Do you have a large number of queues or bindings which churn
at a high rate? You could try setting the ERL_MAX_ETS_TABLES env var -
adding the following to your rabbitmq.conf file should pimp it out for
reasonably extreme running:

SERVER_ERL_ARGS="-env ERL_MAX_ETS_TABLES 90240 -env ERL_MAX_PORTS 40240 +K true +A300 +P512000"

(all one line).

Closing a connection should certainly free up all the resources associated
with it (exclusive queues etc etc), but if you've actually hit some
system limit then it's likely the ErlangVM is in a bad way at that point
and recovery from there is going to be tricky.

We've recently fixed a bug which could cause the ets table exhaustion:
http://www.lshift.net/blog/2010/03/29/on-the-limits-of-concurrency-worker-pools-in-erlang

And the new persister is (needless to say!) even better at resource
management! Depends how bleeding edge you like it really.

> Any thoughts on the behavior I mentioned where the server seemed to continue to accept messages?  I don't see that behavior if I manually stop the server, just in the case of this crash - does that seem at all possible?  I could try to reproduce it locally if that would help.

Yeah, it's possible the queue has died, but that doesn't mean you can't
publish to the exchange - if the queue has died, the exchange won't care
and if the messages are then destined to go to no queues then the exchange
will throw the message away. Setting the mandatory bit on publish will
help here, though maybe at some performance cost.

Matthew