[rabbitmq-discuss] behavior when erlang processes exhausted

Wed May 25 20:31:31 BST 2011

What is the expected behavior of Rabbit when Erlang runs out of
processes? We had an event over the past weekend that went something
like this:

  =ERROR REPORT==== 22-May-2011::17:35:58 ===
  Too many processes

This continued for many hours. Rabbit is configured with 1048576 procs
and with mid-day load is reporting 595400 in use. At the time of this
event, our databases were beginning a prolonged period of fail, and
messages were starting to pile up. I don't have numbers for count of
pending messages at the time of this first log report, but when it
outright crashed 2.5 hours later, the backlog had spiked to more than
2 million and was declining after we fixed the databases and messages
were finally draining.

I have mountains of application and SASL logs and have already poured
through them to highlight notable sections. I'll gladly provide any
data to help diagnose the crash. The final seconds, triggered by a
rabbitmqctl call to list queues, resulted in rex dying and mnesia
exiting.

-Aaron

-- 
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com