[rabbitmq-discuss] rabbit_mgmt_db crashed

Chen, Bin Bin.Chen at kla-tencor.com
Thu Nov 1 20:18:29 GMT 2012


Hi Simon,

We have 4 servers running. The crash did happen frequently.
Please send me the patched management plugin.
I will install it at a proper time.

Thanks,
--Bin Chen

-----Original Message-----
From: Simon MacMullen [mailto:simon at rabbitmq.com] 
Sent: Thursday, November 01, 2012 10:29 AM
To: Chen, Bin
Cc: Discussions about RabbitMQ
Subject: Re: [rabbitmq-discuss] rabbit_mgmt_db crashed

On 31/10/12 18:48, Chen, Bin wrote:
> Hi Simon,
>
> I found the similar crash reports on another RMQ server.
> It had happened a few times.
> However, the memory utilization didn't went up greatly as another 
> server.
>
> Two of the attached zip files are the eval output and report output.
> The other one is the sasl log file containing the crash reports.

Ah, OK. I asked for the eval output to try to understand why that server
was using so much memory - so if it's from a different server using less
memory it doesn't tell me anything.

I'm afraid looking at this I still don't understand any better why the
rabbit_mgmt_db is crashing. But I notice it crashed twice in one day on
that machine - does this happen often for you?

If so, would you be prepared to run a patched management plugin that
could gather some more data when it crashes, so we can investigate
further?

Cheers, Simon

>
> Thanks,
> --Bin Chen
>
>
> -----Original Message-----
> From: Simon MacMullen [mailto:simon at rabbitmq.com]
> Sent: Wednesday, October 31, 2012 4:07 AM
> To: Discussions about RabbitMQ
> Cc: Chen, Bin
> Subject: Re: [rabbitmq-discuss] rabbit_mgmt_db crashed
>
> Hi. I remember you posted a similar stack trace six months ago:
>
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-May/019990.h
> tm
> l
>
> I stared at the code involved for some time but was completely unable 
> to see how that could occur. I don't suppose you have any way to 
> replicate this?
>
> Apart from that, could you post the output of:
>
> rabbitmqctl.bat eval
> "lists:sublist(lists:reverse(lists:sort([{process_info(Pid, memory), 
> Pid, process_info(Pid)} || Pid <- processes()])), 30)."
>
> (all on one line)
>
> and
>
> rabbitmqctl.bat report
>
> somewhere please?
>
> Cheers, Simon
>
> On 31/10/12 00:09, Chen, Bin wrote:
>> Hello,
>>
>> I just noticed a crash report in sasl log of one of our RMQ server.
>>
>> It seems the management plug-in crashed. The management web page 
>> cannot be opened any more.
>>
>> I had this problem before with an old version. Now I am using 2.8.6 
>> 64-bit version on Windows 2008 Server.
>>
>> The RMQ server is still running fine, but the memory utilization is 
>> much higher than other similar servers without such crash.
>>
>> This "bad" server uses about 8 GB memory now, while other normal 
>> servers use only about 500 MB memory.
>>
>> It may eventually hit the high memory water mark soon at 9.6 GB.
>>
>> Here is the complete crash report.
>>
>> =CRASH REPORT==== 29-Oct-2012::23:01:08 ===
>>
>>     crasher:
>>
>>       initial call: rabbit_mgmt_db:init/1
>>
>>       pid: <0.284.0>
>>
>>       registered_name: []
>>
>>       exception exit: {badarith,
>>
>>                           [{rabbit_mgmt_db,rate,5,[]},
>>
>>                            
>> {rabbit_mgmt_db,'-rates/5-lc$^0/1-0-',5,[]},
>>
>>                            
>> {rabbit_mgmt_db,'-rates/5-lc$^1/1-1-',6,[]},
>>
>>                            {rabbit_mgmt_db,rates,5,[]},
>>
>>                            {rabbit_mgmt_db,handle_fine_stat,7,[]},
>>
>>
>> {rabbit_mgmt_db,'-handle_fine_stats/4-lc$^1/1-1-',4,
>>
>>                                []},
>>
>>
>>          {rabbit_mgmt_db,'-handle_event/2-lc$^1/1-0-',4,[]},
>>
>>                            {rabbit_mgmt_db,handle_event,2,[]}]}
>>
>>         in function  gen_server:terminate/6 (gen_server.erl, line 
>> 747)
>>
>>       ancestors: [<0.281.0>,rabbit_mgmt_sup,<0.262.0>]
>>
>>       messages: [{'$gen_cast',
>>
>>                         {event,
>>
>>                             {event,channel_stats,
>>
>>                                 [{pid,<0.12756.396>},
>>
>>                                  {transactional,false},
>>
>>                                  {confirm,false},
>>
>>                                  {consumer_count,0},
>>
>>                                  {messages_unacknowledged,0},
>>
>>                                  {messages_unconfirmed,0},
>>
>>                                  {messages_uncommitted,0},
>>
>>                                  {acks_uncommitted,0},
>>
>>                                  {prefetch_count,1},
>>
>>                                  {client_flow_blocked,false},
>>
>>                                  {channel_queue_stats,[]},
>>
>>                                  {channel_exchange_stats,[]},
>>
>>                                  {channel_queue_exchange_stats,[]}],
>>
>>                                 {1351,576868,812003}}}}]
>>
>>       links: [<0.281.0>]
>>
>>       dictionary: []
>>
>>       trap_exit: false
>>
>>       status: running
>>
>>       heap_size: 6765
>>
>>       stack_size: 24
>>
>>       reductions: 3730467164
>>
>>     neighbours:
>>
>> Here is the output of "rabbitmqctl.bat status"
>>
>> Status of node rabbit at RACK4UNIT001 ...
>>
>> [{pid,4708},
>>
>> {running_applications,
>>
>>        [{rabbitmq_management,"RabbitMQ Management Console","2.8.6"},
>>
>>         {rabbitmq_management_agent,"RabbitMQ Management 
>> Agent","2.8.6"},
>>
>>         {amqp_client,"RabbitMQ AMQP Client","2.8.6"},
>>
>>         {rabbit,"RabbitMQ","2.8.6"},
>>
>>         {os_mon,"CPO  CXC 138 46","2.2.10"},
>>
>>         {sasl,"SASL  CXC 138 11","2.2.1"},
>>
>>         {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.8.6"},
>>
>>         {webmachine,"webmachine","1.9.1-rmq2.8.6-git52e62bc"},
>>
>>         {mochiweb,"MochiMedia Web 
>> Server","2.3.1-rmq2.8.6-gitd541e9a"},
>>
>>         {xmerl,"XML parser","1.3.2"},
>>
>>         {inets,"INETS  CXC 138 49","5.9.1"},
>>
>>         {mnesia,"MNESIA  CXC 138 12","4.7.1"},
>>
>>         {stdlib,"ERTS  CXC 138 10","1.18.2"},
>>
>>         {kernel,"ERTS  CXC 138 10","2.15.2"}]},
>>
>> {os,{win32,nt}},
>>
>> {erlang_version,
>>
>>        "Erlang R15B02 (erts-5.9.2) [64-bit] [smp:12:12] 
>> [async-threads:30]\n"},
>>
>> {memory,
>>
>>        [{total,8512001360},
>>
>>         {processes,7401925315},
>>
>>         {processes_used,7401754972},
>>
>>         {system,1110076045},
>>
>>         {atom,662409},
>>
>>         {atom_used,652416},
>>
>>         {binary,857726336},
>>
>>         {code,17969722},
>>
>>         {ets,4550280}]},
>>
>> {vm_memory_high_watermark,0.4},
>>
>> {vm_memory_limit,10304190873},
>>
>> {disk_free_limit,1000000000},
>>
>> {disk_free,39365828608},
>>
>> {file_descriptors,
>>
>>        [{total_limit,16284},
>>
>>         {total_used,142},
>>
>>         {sockets_limit,14653},
>>
>>         {sockets_used,113}]},
>>
>> {processes,[{limit,1048576},{used,5986}]},
>>
>> {run_queue,0},
>>
>> {uptime,104350}]
>>
>> ...done.
>>
>> Best regards,
>>
>> --Bin Chen
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>


--
Simon MacMullen
RabbitMQ, VMware


More information about the rabbitmq-discuss mailing list