[rabbitmq-discuss] RabbitMQ 3.1.5 crashed

[Simon MacMullen]

Thanks.

The VM crashed because it couldn't allocate memory.

Normally the memory monitor should detect that RabbitMQ is starting to 
use too much memory and prevent further publishing before reaching this 
point (see http://www.rabbitmq.com/memory.html#memsup). However, you hit 
a long-standing bug that made this less likely to happen.

When running RabbitMQ on a 32 bit Erlang VM on a 64 bit OS as you are, 
the memory monitor detects (in your case) 12GB physical memory in the 
OS, but then adjusts that for the smaller 32 bit address space. 
Unfortunately in versions of RabbitMQ prior to 3.2.1 running on Linux it 
adjusts the memory limit down to 4GB - but in fact the kernel will 
generally only make 2GB addressable.

So the memory alarm will only go off when really quite close to the 
limit. And in this 32 bit VM on 64 bit OS that's especially deadly - 
normally when you run out of physical memory you start to swap, which is 
not great but not disastrous - but when you run out of address space you 
hit a brick wall and die.

In fact I think I'm going to beef up that warning message, it should say 
"crashes are much more likely" or similar.

So there are various things you could do about this (upgrade RabbitMQ, 
reduce vm_memory_high_watermark) but by far the best thing would be to 
start using a 64 bit Erlang build.

Cheers, Simon

On 18/12/13 23:31, Kapil Goyal wrote:
> Thanks Simon,
>
> I am attaching more logs (zipped) from that day.
>
> Crash dump is available at http://goo.gl/iB5Z8Y.
>
> Thanks
> Kapil
>
> ----- Original Message -----
> From: "Simon MacMullen" <simon at rabbitmq.com>
> To: "Discussions about RabbitMQ" <rabbitmq-discuss at lists.rabbitmq.com>
> Cc: "Kapil Goyal" <goyalk at vmware.com>
> Sent: Tuesday, December 17, 2013 3:39:55 AM
> Subject: Re: [rabbitmq-discuss] RabbitMQ 3.1.5 crashed
>
> So what I see from the logs is that a connection crashed since it
> received connection.close and then found that the TCP connection was
> already closed when it tried to send connection.close-ok in response.
>
> That then caused the queue collector for that connection (component
> which deletes exclusive queues when a connection closes) to crash.
>
> So that's some noise in the logs, I think that could be handled better -
> but I don't see anything there that would amount to the server itself
> crashing. Can you post complete logs somewhere? Probably more useful
> than the crash dump.
>
> Cheers, Simon
>
> On 16/12/2013 23:37, Kapil Goyal wrote:
>> Hi,
>>
>> A rabbitmq server in one of our setups crashed recently. Can you please
>> take a look at the attached rabbit and rabbit-sasl logs and help me
>> understand the issue?
>>
>> The logs attached start from around a minute before and until the crash.
>>
>> I also have a 17M crash dump file. If you'd like to look at it, can I
>> send it as an email attachment or you'd prefer another way?
>>
>>
>> Thanks
>> Kapil
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://urldefense.proofpoint.com/v1/url?u=https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=JF6BIRiITniYYZhbhzlgNvj%2FeB0%2BYbRBSr0f8NG91qs%3D%0A&m=Z6q%2FpZ9j4Nys6NUpFt3IweKfXsGwzUAXjgvVXmGatz4%3D%0A&s=a92d49282fa22b791fd4ee337e0d46e8d8dc9bea47f5729bd1d35a2305a2a117
>>


-- 
Simon MacMullen
RabbitMQ, Pivotal