[rabbitmq-discuss] RabbitMQ crash

Tue Nov 20 15:10:45 GMT 2012

Thanks for the detailed insight. I am attaching the erl_crash.dump here, if
it helps. The crash happened on Nov 19 around 14:50 hrs.

-Mahesh

On Tue, Nov 20, 2012 at 7:43 PM, Tim Watson <tim at rabbitmq.com> wrote:

> Also....
>
> It is often possible to pinpoint memory allocation culprits from the
> output of  "rabbitmqctl report", e.g. if there are many channels consuming
> RAM. You might also consider using the latest version of the management
> plugin, which displays memory allocation details nicely. It is, of course,
> possible that another application on the server is chewing up all available
> RAM.
>
> Cheers,
> Tim
>
> On 20 Nov 2012, at 14:03, Tim Watson wrote:
>
> > And the rabbit logs? Do they contain any memory alarm notifications?
> This should've kicked in when the high watermark was reached (around 9.6Gb
> for a 64bit executable on Windows running R15B01) but that would cause
> paging and/or memory based flow control to kick in in the first instance.
> Your crash dump indicates that the situation worsened far beyond that
> point, so we should see *something* in the logs earlier on.
> >
> > Also what about the windows system log - can you see anything in there?
> What else is running on this machine at the same time? Are there signs of
> all memory being exhausted? Windows does put limits on per-process virtual
> memory space (even for 64bit processes depending on the presence and value
> of the IMAGE_FILE_LARGE_ADDRESS_AWARE flag for the image) so it's entirely
> possible that the vm was unable to allocate this tiny amount of memory
> despite your 24GB of installed RAM.
> >
> > As I said the rabbit logs and windows system logs should contain some
> useful information. It might also help to explain what kind of load the
> system was under when this happened.
> >
> > Cheers,
> > Tim
> >
> > On 20 Nov 2012, at 12:51, Mahesh Viraktamath wrote:
> >
> >> It is a Windows 2008 Server R2 with RAM of 24 GB.
> >>
> >> Here's the output of rabbitmqctl environment:
> >>
> >> Application environment of node 'rabbit at 347943-rab' ...
> >> [{auth_backends,[rabbit_auth_backend_internal]},
> >> {auth_mechanisms,['PLAIN','AMQPLAIN']},
> >> {backing_queue_module,rabbit_variable_queue},
> >> {cluster_nodes,[]},
> >> {collect_statistics,fine},
> >> {collect_statistics_interval,5000},
> >> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
> >> {default_user,<<"guest">>},
> >> {default_user_tags,[administrator]},
> >> {default_vhost,<<"/">>},
> >> {delegate_count,16},
> >> {disk_free_limit,1000000000},
> >>
> {error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit at 347943-rab.log
> "}},
> >> {frame_max,131072},
> >> {hipe_compile,false},
> >> {included_applications,[]},
> >> {log_levels,[{connection,info}]},
> >> {msg_store_file_size_limit,16777216},
> >> {msg_store_index_module,rabbit_msg_store_ets_index},
> >> {queue_index_max_journal_entries,262144},
> >>
> {sasl_error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit at 347943-rab-sasl.log
> "}},
> >> {server_properties,[]},
> >> {ssl_cert_login_from,distinguished_name},
> >> {ssl_listeners,[]},
> >> {ssl_options,[]},
> >> {tcp_listen_options,[binary,
> >>                      {packet,raw},
> >>                      {reuseaddr,true},
> >>                      {backlog,128},
> >>                      {nodelay,true},
> >>                      {linger,{true,0}},
> >>                      {exit_on_close,false}]},
> >> {tcp_listeners,[5672]},
> >> {trace_vhosts,[]},
> >> {vm_memory_high_watermark,0.4}]
> >> ...done.
> >>
> >>
> >> --Mahesh
> >>
> >>
> >> On Tue, Nov 20, 2012 at 2:50 PM, Tim Watson <watson.timothy at gmail.com>
> wrote:
> >> Looks like your system completely ran out of available memory and swap
> space at which point alloc Started to fail. Rabbit tries hard to avoid this
> (by paging messages to disk and applying memory based flow control to
> throttle senders) but if the whole system completely runs out of ram (and
> swap space is exhausted) then no program will be able to continue running
> if it needs dynamically allocated memory that is not available.
> >>
> >> What does your environment look like (run rabbitmqctl environment) such
> as os, available memory, swap space, etc?
> >>
> >> Have you configured the memory high watermark? There should be some
> alarms going off in the rabbit logs before the emulator crashes if
> memory/swap has really been exhausted.
> >>
> >> On 20 Nov 2012, at 06:53, Mahesh Viraktamath <yuva670 at gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> We are running RabbitMQ 2.8.4 on production for past 2 months, it
> crashed yesterday and left a huge crash dump (about 5 MB). The log starts
> like this:
> >>>
> >>> =erl_crash_dump:0.1
> >>> Mon Nov 19 14:51:22 2012
> >>> Slogan: temp_alloc: Cannot allocate 623672 bytes of memory (of type
> "tmp_heap").
> >>> System version: Erlang R15B01 (erts-5.9.1) [64-bit] [smp:12:12]
> [async-threads:30]
> >>> Compiled: Sun Apr  1 19:26:21 2012
> >>>
> >>> If this is not enough, I can attach the whole dump.
> >>>
> >>> How can I avoid the crash in future, please advice.
> >>>
> >>>
> >>> -Mahesh
> >>> _______________________________________________
> >>> rabbitmq-discuss mailing list
> >>> rabbitmq-discuss at lists.rabbitmq.com
> >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >> _______________________________________________
> >> rabbitmq-discuss mailing list
> >> rabbitmq-discuss at lists.rabbitmq.com
> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >>
> >> _______________________________________________
> >> rabbitmq-discuss mailing list
> >> rabbitmq-discuss at lists.rabbitmq.com
> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121120/50205496/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: erl_crash.dump
Type: application/octet-stream
Size: 5339053 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121120/50205496/attachment-0001.obj>