[rabbitmq-discuss] RabbitMQ crash

Tim Watson watson.timothy at gmail.com
Tue Nov 20 16:49:19 GMT 2012


All the useful information in the crash dump is in the heading you've already sent I'm afraid. What I need to see is the rabbit log or the output of rabbitmqctl report while the server was under heavy load I realise the latter isn't possible to produce posthumously! :)

On 20 Nov 2012, at 15:10, Mahesh Viraktamath <yuva670 at gmail.com> wrote:

> Thanks for the detailed insight. I am attaching the erl_crash.dump here, if it helps. The crash happened on Nov 19 around 14:50 hrs.
> 
> 
> -Mahesh
> 
> 
> On Tue, Nov 20, 2012 at 7:43 PM, Tim Watson <tim at rabbitmq.com> wrote:
> Also....
> 
> It is often possible to pinpoint memory allocation culprits from the output of  "rabbitmqctl report", e.g. if there are many channels consuming RAM. You might also consider using the latest version of the management plugin, which displays memory allocation details nicely. It is, of course, possible that another application on the server is chewing up all available RAM.
> 
> Cheers,
> Tim
> 
> On 20 Nov 2012, at 14:03, Tim Watson wrote:
> 
> > And the rabbit logs? Do they contain any memory alarm notifications? This should've kicked in when the high watermark was reached (around 9.6Gb for a 64bit executable on Windows running R15B01) but that would cause paging and/or memory based flow control to kick in in the first instance. Your crash dump indicates that the situation worsened far beyond that point, so we should see *something* in the logs earlier on.
> >
> > Also what about the windows system log - can you see anything in there? What else is running on this machine at the same time? Are there signs of all memory being exhausted? Windows does put limits on per-process virtual memory space (even for 64bit processes depending on the presence and value of the IMAGE_FILE_LARGE_ADDRESS_AWARE flag for the image) so it's entirely possible that the vm was unable to allocate this tiny amount of memory despite your 24GB of installed RAM.
> >
> > As I said the rabbit logs and windows system logs should contain some useful information. It might also help to explain what kind of load the system was under when this happened.
> >
> > Cheers,
> > Tim
> >
> > On 20 Nov 2012, at 12:51, Mahesh Viraktamath wrote:
> >
> >> It is a Windows 2008 Server R2 with RAM of 24 GB.
> >>
> >> Here's the output of rabbitmqctl environment:
> >>
> >> Application environment of node 'rabbit at 347943-rab' ...
> >> [{auth_backends,[rabbit_auth_backend_internal]},
> >> {auth_mechanisms,['PLAIN','AMQPLAIN']},
> >> {backing_queue_module,rabbit_variable_queue},
> >> {cluster_nodes,[]},
> >> {collect_statistics,fine},
> >> {collect_statistics_interval,5000},
> >> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
> >> {default_user,<<"guest">>},
> >> {default_user_tags,[administrator]},
> >> {default_vhost,<<"/">>},
> >> {delegate_count,16},
> >> {disk_free_limit,1000000000},
> >> {error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit at 347943-rab.log"}},
> >> {frame_max,131072},
> >> {hipe_compile,false},
> >> {included_applications,[]},
> >> {log_levels,[{connection,info}]},
> >> {msg_store_file_size_limit,16777216},
> >> {msg_store_index_module,rabbit_msg_store_ets_index},
> >> {queue_index_max_journal_entries,262144},
> >> {sasl_error_logger,{file,"C:/Users/Administrator/AppData/Roaming/RabbitMQ/log/rabbit at 347943-rab-sasl.log"}},
> >> {server_properties,[]},
> >> {ssl_cert_login_from,distinguished_name},
> >> {ssl_listeners,[]},
> >> {ssl_options,[]},
> >> {tcp_listen_options,[binary,
> >>                      {packet,raw},
> >>                      {reuseaddr,true},
> >>                      {backlog,128},
> >>                      {nodelay,true},
> >>                      {linger,{true,0}},
> >>                      {exit_on_close,false}]},
> >> {tcp_listeners,[5672]},
> >> {trace_vhosts,[]},
> >> {vm_memory_high_watermark,0.4}]
> >> ...done.
> >>
> >>
> >> --Mahesh
> >>
> >>
> >> On Tue, Nov 20, 2012 at 2:50 PM, Tim Watson <watson.timothy at gmail.com> wrote:
> >> Looks like your system completely ran out of available memory and swap space at which point alloc Started to fail. Rabbit tries hard to avoid this (by paging messages to disk and applying memory based flow control to throttle senders) but if the whole system completely runs out of ram (and swap space is exhausted) then no program will be able to continue running if it needs dynamically allocated memory that is not available.
> >>
> >> What does your environment look like (run rabbitmqctl environment) such as os, available memory, swap space, etc?
> >>
> >> Have you configured the memory high watermark? There should be some alarms going off in the rabbit logs before the emulator crashes if memory/swap has really been exhausted.
> >>
> >> On 20 Nov 2012, at 06:53, Mahesh Viraktamath <yuva670 at gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> We are running RabbitMQ 2.8.4 on production for past 2 months, it crashed yesterday and left a huge crash dump (about 5 MB). The log starts like this:
> >>>
> >>> =erl_crash_dump:0.1
> >>> Mon Nov 19 14:51:22 2012
> >>> Slogan: temp_alloc: Cannot allocate 623672 bytes of memory (of type "tmp_heap").
> >>> System version: Erlang R15B01 (erts-5.9.1) [64-bit] [smp:12:12] [async-threads:30]
> >>> Compiled: Sun Apr  1 19:26:21 2012
> >>>
> >>> If this is not enough, I can attach the whole dump.
> >>>
> >>> How can I avoid the crash in future, please advice.
> >>>
> >>>
> >>> -Mahesh
> >>> _______________________________________________
> >>> rabbitmq-discuss mailing list
> >>> rabbitmq-discuss at lists.rabbitmq.com
> >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >> _______________________________________________
> >> rabbitmq-discuss mailing list
> >> rabbitmq-discuss at lists.rabbitmq.com
> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >>
> >> _______________________________________________
> >> rabbitmq-discuss mailing list
> >> rabbitmq-discuss at lists.rabbitmq.com
> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> <erl_crash.dump>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121120/933e81ea/attachment.htm>


More information about the rabbitmq-discuss mailing list