Thanks for the detailed insight. I am attaching the erl_crash.dump here, if it helps. The crash happened on Nov 19 around 14:50 hrs.<div><br></div><div><br></div><div>-Mahesh</div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Tue, Nov 20, 2012 at 7:43 PM, Tim Watson <span dir="ltr"><<a href="mailto:tim@rabbitmq.com" target="_blank">tim@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Also....<br>
<br>
It is often possible to pinpoint memory allocation culprits from the output of "rabbitmqctl report", e.g. if there are many channels consuming RAM. You might also consider using the latest version of the management plugin, which displays memory allocation details nicely. It is, of course, possible that another application on the server is chewing up all available RAM.<br>
<br>
Cheers,<br>
Tim<br>
<div class="HOEnZb"><div class="h5"><br>
On 20 Nov 2012, at 14:03, Tim Watson wrote:<br>
<br>
> And the rabbit logs? Do they contain any memory alarm notifications? This should've kicked in when the high watermark was reached (around 9.6Gb for a 64bit executable on Windows running R15B01) but that would cause paging and/or memory based flow control to kick in in the first instance. Your crash dump indicates that the situation worsened far beyond that point, so we should see *something* in the logs earlier on.<br>
><br>
> Also what about the windows system log - can you see anything in there? What else is running on this machine at the same time? Are there signs of all memory being exhausted? Windows does put limits on per-process virtual memory space (even for 64bit processes depending on the presence and value of the IMAGE_FILE_LARGE_ADDRESS_AWARE flag for the image) so it's entirely possible that the vm was unable to allocate this tiny amount of memory despite your 24GB of installed RAM.<br>
><br>
> As I said the rabbit logs and windows system logs should contain some useful information. It might also help to explain what kind of load the system was under when this happened.<br>
><br>
> Cheers,<br>
> Tim<br>
><br>
> On 20 Nov 2012, at 12:51, Mahesh Viraktamath wrote:<br>
><br>
>> It is a Windows 2008 Server R2 with RAM of 24 GB.<br>
>><br>
>> Here's the output of rabbitmqctl environment:<br>
>><br>
>> Application environment of node 'rabbit@347943-rab' ...<br>
>> [{auth_backends,[rabbit_auth_backend_internal]},<br>
>> {auth_mechanisms,['PLAIN','AMQPLAIN']},<br>
>> {backing_queue_module,rabbit_variable_queue},<br>
>> {cluster_nodes,[]},<br>
>> {collect_statistics,fine},<br>
>> {collect_statistics_interval,5000},<br>
>> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},<br>
>> {default_user,<<"guest">>},<br>
>> {default_user_tags,[administrator]},<br>
>> {default_vhost,<<"/">>},<br>
>> {delegate_count,16},<br>
>> {disk_free_limit,1000000000},<br>
>> {error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit@347943-rab.log"}},<br>
>> {frame_max,131072},<br>
>> {hipe_compile,false},<br>
>> {included_applications,[]},<br>
>> {log_levels,[{connection,info}]},<br>
>> {msg_store_file_size_limit,16777216},<br>
>> {msg_store_index_module,rabbit_msg_store_ets_index},<br>
>> {queue_index_max_journal_entries,262144},<br>
>> {sasl_error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit@347943-rab-sasl.log"}},<br>
>> {server_properties,[]},<br>
>> {ssl_cert_login_from,distinguished_name},<br>
>> {ssl_listeners,[]},<br>
>> {ssl_options,[]},<br>
>> {tcp_listen_options,[binary,<br>
>> {packet,raw},<br>
>> {reuseaddr,true},<br>
>> {backlog,128},<br>
>> {nodelay,true},<br>
>> {linger,{true,0}},<br>
>> {exit_on_close,false}]},<br>
>> {tcp_listeners,[5672]},<br>
>> {trace_vhosts,[]},<br>
>> {vm_memory_high_watermark,0.4}]<br>
>> ...done.<br>
>><br>
>><br>
>> --Mahesh<br>
>><br>
>><br>
>> On Tue, Nov 20, 2012 at 2:50 PM, Tim Watson <<a href="mailto:watson.timothy@gmail.com">watson.timothy@gmail.com</a>> wrote:<br>
>> Looks like your system completely ran out of available memory and swap space at which point alloc Started to fail. Rabbit tries hard to avoid this (by paging messages to disk and applying memory based flow control to throttle senders) but if the whole system completely runs out of ram (and swap space is exhausted) then no program will be able to continue running if it needs dynamically allocated memory that is not available.<br>
>><br>
>> What does your environment look like (run rabbitmqctl environment) such as os, available memory, swap space, etc?<br>
>><br>
>> Have you configured the memory high watermark? There should be some alarms going off in the rabbit logs before the emulator crashes if memory/swap has really been exhausted.<br>
>><br>
>> On 20 Nov 2012, at 06:53, Mahesh Viraktamath <<a href="mailto:yuva670@gmail.com">yuva670@gmail.com</a>> wrote:<br>
>><br>
>>> Hi,<br>
>>><br>
>>> We are running RabbitMQ 2.8.4 on production for past 2 months, it crashed yesterday and left a huge crash dump (about 5 MB). The log starts like this:<br>
>>><br>
>>> =erl_crash_dump:0.1<br>
>>> Mon Nov 19 14:51:22 2012<br>
>>> Slogan: temp_alloc: Cannot allocate 623672 bytes of memory (of type "tmp_heap").<br>
>>> System version: Erlang R15B01 (erts-5.9.1) [64-bit] [smp:12:12] [async-threads:30]<br>
>>> Compiled: Sun Apr 1 19:26:21 2012<br>
>>><br>
>>> If this is not enough, I can attach the whole dump.<br>
>>><br>
>>> How can I avoid the crash in future, please advice.<br>
>>><br>
>>><br>
>>> -Mahesh<br>
>>> _______________________________________________<br>
>>> rabbitmq-discuss mailing list<br>
>>> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
>>> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
>> _______________________________________________<br>
>> rabbitmq-discuss mailing list<br>
>> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
>> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
>><br>
>> _______________________________________________<br>
>> rabbitmq-discuss mailing list<br>
>> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
>> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
><br>
> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div>