[rabbitmq-discuss] RabbitMQ crashes hard when it runs out of memory

[Matthew Sackman]

Hi Steve,

On Fri, Nov 06, 2009 at 12:57:33PM -0800, Stephen Day wrote:
> On Fri, Nov 6, 2009 at 2:43 AM, Matthew Sackman <matthew at lshift.net> wrote:
> However, with rabbit's current memory problems, I wouldn't run it with more
> than 10 queues, let alone millions ;). I will give the java client test a
> try.

Queues themselves don't take much memory - it's just a process and a
little state. I'm seeing about 35KB for a queue with nothing in it. It's
the number and size of messages that are the issue. We tend to encourage
users to make sure that the bottleneck is at the producing side, not the
consuming side, to prevent queues growing much in Rabbit. In such a
setup, many thousands of queues is very possible. However, people using
Rabbit as a buffering / decoupling mechanism to cope with load spikes
between different systems is a very common and important use case and
this should get much much better once the new resource management and
persister finally lands.

> This was my initial thought as well. However, by the time the alarm goes
> off, it is often too late for this to stop rabbit from crashing. For
> example, the default memory alarm is set to 0.95.

Right, but it's not hardcoded at 0.95. Feel free to lower it
substantially.

> I would say the correct pattern would be something like python's MemoryError
> or handling of a malloc failure in C. Is there an exception that is thrown,
> where a garbage collect can be run?

Sure, but what exactly can you do in that situation? The malloc is
failing during a GC run, because of the generational copying GC, so it
doesn't really make much difference whether you catch the exception or
not - you still can't allocate new memory, which makes life somewhat
challenging, especially in the middle of a GC run, where processes are
suspended.

Our resource management is getting much better, and is something we're
working hard on improving. Sadly though, we're not going to give out
ETAs only to break them, though we understand how fun that is for all ;)

Matthew