[rabbitmq-discuss] RabbitMQ crash

[Tim Watson]

And the rabbit logs? Do they contain any memory alarm notifications? This should've kicked in when the high watermark was reached (around 9.6Gb for a 64bit executable on Windows running R15B01) but that would cause paging and/or memory based flow control to kick in in the first instance. Your crash dump indicates that the situation worsened far beyond that point, so we should see *something* in the logs earlier on.

Also what about the windows system log - can you see anything in there? What else is running on this machine at the same time? Are there signs of all memory being exhausted? Windows does put limits on per-process virtual memory space (even for 64bit processes depending on the presence and value of the IMAGE_FILE_LARGE_ADDRESS_AWARE flag for the image) so it's entirely possible that the vm was unable to allocate this tiny amount of memory despite your 24GB of installed RAM.

As I said the rabbit logs and windows system logs should contain some useful information. It might also help to explain what kind of load the system was under when this happened. 

Cheers,
Tim

On 20 Nov 2012, at 12:51, Mahesh Viraktamath wrote:

> It is a Windows 2008 Server R2 with RAM of 24 GB.
> 
> Here's the output of rabbitmqctl environment:
> 
> Application environment of node 'rabbit at 347943-rab' ...
> [{auth_backends,[rabbit_auth_backend_internal]},
>  {auth_mechanisms,['PLAIN','AMQPLAIN']},
>  {backing_queue_module,rabbit_variable_queue},
>  {cluster_nodes,[]},
>  {collect_statistics,fine},
>  {collect_statistics_interval,5000},
>  {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
>  {default_user,<<"guest">>},
>  {default_user_tags,[administrator]},
>  {default_vhost,<<"/">>},
>  {delegate_count,16},
>  {disk_free_limit,1000000000},
>  {error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit at 347943-rab.log"}},
>  {frame_max,131072},
>  {hipe_compile,false},
>  {included_applications,[]},
>  {log_levels,[{connection,info}]},
>  {msg_store_file_size_limit,16777216},
>  {msg_store_index_module,rabbit_msg_store_ets_index},
>  {queue_index_max_journal_entries,262144},
>  {sasl_error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit at 347943-rab-sasl.log"}},
>  {server_properties,[]},
>  {ssl_cert_login_from,distinguished_name},
>  {ssl_listeners,[]},
>  {ssl_options,[]},
>  {tcp_listen_options,[binary,
>                       {packet,raw},
>                       {reuseaddr,true},
>                       {backlog,128},
>                       {nodelay,true},
>                       {linger,{true,0}},
>                       {exit_on_close,false}]},
>  {tcp_listeners,[5672]},
>  {trace_vhosts,[]},
>  {vm_memory_high_watermark,0.4}]
> ...done.
> 
> 
> --Mahesh
> 
> 
> On Tue, Nov 20, 2012 at 2:50 PM, Tim Watson <watson.timothy at gmail.com> wrote:
> Looks like your system completely ran out of available memory and swap space at which point alloc Started to fail. Rabbit tries hard to avoid this (by paging messages to disk and applying memory based flow control to throttle senders) but if the whole system completely runs out of ram (and swap space is exhausted) then no program will be able to continue running if it needs dynamically allocated memory that is not available.
> 
> What does your environment look like (run rabbitmqctl environment) such as os, available memory, swap space, etc?
> 
> Have you configured the memory high watermark? There should be some alarms going off in the rabbit logs before the emulator crashes if memory/swap has really been exhausted.
> 
> On 20 Nov 2012, at 06:53, Mahesh Viraktamath <yuva670 at gmail.com> wrote:
> 
> > Hi,
> >
> > We are running RabbitMQ 2.8.4 on production for past 2 months, it crashed yesterday and left a huge crash dump (about 5 MB). The log starts like this:
> >
> > =erl_crash_dump:0.1
> > Mon Nov 19 14:51:22 2012
> > Slogan: temp_alloc: Cannot allocate 623672 bytes of memory (of type "tmp_heap").
> > System version: Erlang R15B01 (erts-5.9.1) [64-bit] [smp:12:12] [async-threads:30]
> > Compiled: Sun Apr  1 19:26:21 2012
> >
> > If this is not enough, I can attach the whole dump.
> >
> > How can I avoid the crash in future, please advice.
> >
> >
> > -Mahesh
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss