[rabbitmq-discuss] OOM kill

[Jason McIntosh]

Simon:

Is it messages or connections that it blocks?  I thought it would block new
connections but still allow existing connections to publish messages?  At
what point does rabbit start paging to disk on the messages?  Is that at
50% of the vm memory threshold?

Jason


On Fri, Mar 7, 2014 at 11:45 AM, Simon MacMullen <simon at rabbitmq.com> wrote:

> On 07/03/2014 5:16PM, Dmitry Andrianov wrote:
>
>> Hello.
>>
>
> Hi.
>
>
>  We are trying to load-test RabbitMQ server in different configurations
>> on Amazon EC2 node.
>> Most of our tests end with Linux OOM killer intervening and killing
>> Rabbit.
>> That is something I cannot really understand especially given that it is
>> reproducible even with vm_memory_high_watermark set to 0.2 and no other
>> processes running on that box.
>> So if someone could shed some light onto that issue it would help a lot.
>>
>> Below the status response not long before the process was killed:
>>
>
> <snip>
>
>
>  Couple of strange things there are:
>>
>> 1. {vm_memory_limit,804643635} but still memory {total,1984625336}. How
>> is that possible? https://www.rabbitmq.com/memory.html says that erlang
>> process can take twice the configured size so I expected that but it is
>> definitely more than twice.
>>
>
> The only ways RabbitMQ has of preventing memory use from increasing are to
> do with messages - when the memory alarm goes off it will stop accepting
> new messages, and before that point it will start trying to reduce memory
> use by paging messages out to disc.
>
> Normally, messages are the biggest user of memory in RabbitMQ, so this
> approach works OK.
>
> However, your test ends up causing RabbitMQ to end up using the majority
> of its memory in connection processes - you have 11k connections open, at
> about 120kB each.
>
> We don't prevent RabbitMQ from accepting new connections when the memory
> alarm goes off since our main worry is messages - and those connections
> could be intending to consume messages and thus reduce memory pressure.
>
> So I guess you might want to reduce the ulimit, so that RabbitMQ runs out
> of file descriptors before it runs out of memory (when it runs out of FDs
> it *will* stop accepting network connections gracefully).
>
>
>  2. {plugins,-44730984} - how this one is possible?
>>
>
> That's a good question!
>
> That value is calculated from (memory used by all plugins including
> management) - (memory used by the management database). So somehow the
> memory counter managed to determine that "all plugins" used less memory
> than just the management plugin. I'll look into that.
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



-- 
Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140307/f70924e5/attachment.html>