[rabbitmq-discuss] OOM kill

[Simon MacMullen]

On 07/03/2014 5:16PM, Dmitry Andrianov wrote:
> Hello.

Hi.

> We are trying to load-test RabbitMQ server in different configurations
> on Amazon EC2 node.
> Most of our tests end with Linux OOM killer intervening and killing Rabbit.
> That is something I cannot really understand especially given that it is
> reproducible even with vm_memory_high_watermark set to 0.2 and no other
> processes running on that box.
> So if someone could shed some light onto that issue it would help a lot.
>
> Below the status response not long before the process was killed:

<snip>

> Couple of strange things there are:
>
> 1. {vm_memory_limit,804643635} but still memory {total,1984625336}. How
> is that possible? https://www.rabbitmq.com/memory.html says that erlang
> process can take twice the configured size so I expected that but it is
> definitely more than twice.

The only ways RabbitMQ has of preventing memory use from increasing are 
to do with messages - when the memory alarm goes off it will stop 
accepting new messages, and before that point it will start trying to 
reduce memory use by paging messages out to disc.

Normally, messages are the biggest user of memory in RabbitMQ, so this 
approach works OK.

However, your test ends up causing RabbitMQ to end up using the majority 
of its memory in connection processes - you have 11k connections open, 
at about 120kB each.

We don't prevent RabbitMQ from accepting new connections when the memory 
alarm goes off since our main worry is messages - and those connections 
could be intending to consume messages and thus reduce memory pressure.

So I guess you might want to reduce the ulimit, so that RabbitMQ runs 
out of file descriptors before it runs out of memory (when it runs out 
of FDs it *will* stop accepting network connections gracefully).

> 2. {plugins,-44730984} - how this one is possible?

That's a good question!

That value is calculated from (memory used by all plugins including 
management) - (memory used by the management database). So somehow the 
memory counter managed to determine that "all plugins" used less memory 
than just the management plugin. I'll look into that.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal