[rabbitmq-discuss] rabbit_mgmt_db crashed
Simon MacMullen
simon at rabbitmq.com
Thu Nov 1 17:28:47 GMT 2012
On 31/10/12 18:48, Chen, Bin wrote:
> Hi Simon,
>
> I found the similar crash reports on another RMQ server.
> It had happened a few times.
> However, the memory utilization didn't went up greatly as another
> server.
>
> Two of the attached zip files are the eval output and report output.
> The other one is the sasl log file containing the crash reports.
Ah, OK. I asked for the eval output to try to understand why that server
was using so much memory - so if it's from a different server using less
memory it doesn't tell me anything.
I'm afraid looking at this I still don't understand any better why the
rabbit_mgmt_db is crashing. But I notice it crashed twice in one day on
that machine - does this happen often for you?
If so, would you be prepared to run a patched management plugin that
could gather some more data when it crashes, so we can investigate further?
Cheers, Simon
>
> Thanks,
> --Bin Chen
>
>
> -----Original Message-----
> From: Simon MacMullen [mailto:simon at rabbitmq.com]
> Sent: Wednesday, October 31, 2012 4:07 AM
> To: Discussions about RabbitMQ
> Cc: Chen, Bin
> Subject: Re: [rabbitmq-discuss] rabbit_mgmt_db crashed
>
> Hi. I remember you posted a similar stack trace six months ago:
>
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-May/019990.htm
> l
>
> I stared at the code involved for some time but was completely unable to
> see how that could occur. I don't suppose you have any way to replicate
> this?
>
> Apart from that, could you post the output of:
>
> rabbitmqctl.bat eval
> "lists:sublist(lists:reverse(lists:sort([{process_info(Pid, memory),
> Pid, process_info(Pid)} || Pid <- processes()])), 30)."
>
> (all on one line)
>
> and
>
> rabbitmqctl.bat report
>
> somewhere please?
>
> Cheers, Simon
>
> On 31/10/12 00:09, Chen, Bin wrote:
>> Hello,
>>
>> I just noticed a crash report in sasl log of one of our RMQ server.
>>
>> It seems the management plug-in crashed. The management web page
>> cannot be opened any more.
>>
>> I had this problem before with an old version. Now I am using 2.8.6
>> 64-bit version on Windows 2008 Server.
>>
>> The RMQ server is still running fine, but the memory utilization is
>> much higher than other similar servers without such crash.
>>
>> This "bad" server uses about 8 GB memory now, while other normal
>> servers use only about 500 MB memory.
>>
>> It may eventually hit the high memory water mark soon at 9.6 GB.
>>
>> Here is the complete crash report.
>>
>> =CRASH REPORT==== 29-Oct-2012::23:01:08 ===
>>
>> crasher:
>>
>> initial call: rabbit_mgmt_db:init/1
>>
>> pid: <0.284.0>
>>
>> registered_name: []
>>
>> exception exit: {badarith,
>>
>> [{rabbit_mgmt_db,rate,5,[]},
>>
>> {rabbit_mgmt_db,'-rates/5-lc$^0/1-0-',5,[]},
>>
>> {rabbit_mgmt_db,'-rates/5-lc$^1/1-1-',6,[]},
>>
>> {rabbit_mgmt_db,rates,5,[]},
>>
>> {rabbit_mgmt_db,handle_fine_stat,7,[]},
>>
>>
>> {rabbit_mgmt_db,'-handle_fine_stats/4-lc$^1/1-1-',4,
>>
>> []},
>>
>>
>> {rabbit_mgmt_db,'-handle_event/2-lc$^1/1-0-',4,[]},
>>
>> {rabbit_mgmt_db,handle_event,2,[]}]}
>>
>> in function gen_server:terminate/6 (gen_server.erl, line 747)
>>
>> ancestors: [<0.281.0>,rabbit_mgmt_sup,<0.262.0>]
>>
>> messages: [{'$gen_cast',
>>
>> {event,
>>
>> {event,channel_stats,
>>
>> [{pid,<0.12756.396>},
>>
>> {transactional,false},
>>
>> {confirm,false},
>>
>> {consumer_count,0},
>>
>> {messages_unacknowledged,0},
>>
>> {messages_unconfirmed,0},
>>
>> {messages_uncommitted,0},
>>
>> {acks_uncommitted,0},
>>
>> {prefetch_count,1},
>>
>> {client_flow_blocked,false},
>>
>> {channel_queue_stats,[]},
>>
>> {channel_exchange_stats,[]},
>>
>> {channel_queue_exchange_stats,[]}],
>>
>> {1351,576868,812003}}}}]
>>
>> links: [<0.281.0>]
>>
>> dictionary: []
>>
>> trap_exit: false
>>
>> status: running
>>
>> heap_size: 6765
>>
>> stack_size: 24
>>
>> reductions: 3730467164
>>
>> neighbours:
>>
>> Here is the output of "rabbitmqctl.bat status"
>>
>> Status of node rabbit at RACK4UNIT001 ...
>>
>> [{pid,4708},
>>
>> {running_applications,
>>
>> [{rabbitmq_management,"RabbitMQ Management Console","2.8.6"},
>>
>> {rabbitmq_management_agent,"RabbitMQ Management
>> Agent","2.8.6"},
>>
>> {amqp_client,"RabbitMQ AMQP Client","2.8.6"},
>>
>> {rabbit,"RabbitMQ","2.8.6"},
>>
>> {os_mon,"CPO CXC 138 46","2.2.10"},
>>
>> {sasl,"SASL CXC 138 11","2.2.1"},
>>
>> {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.8.6"},
>>
>> {webmachine,"webmachine","1.9.1-rmq2.8.6-git52e62bc"},
>>
>> {mochiweb,"MochiMedia Web Server","2.3.1-rmq2.8.6-gitd541e9a"},
>>
>> {xmerl,"XML parser","1.3.2"},
>>
>> {inets,"INETS CXC 138 49","5.9.1"},
>>
>> {mnesia,"MNESIA CXC 138 12","4.7.1"},
>>
>> {stdlib,"ERTS CXC 138 10","1.18.2"},
>>
>> {kernel,"ERTS CXC 138 10","2.15.2"}]},
>>
>> {os,{win32,nt}},
>>
>> {erlang_version,
>>
>> "Erlang R15B02 (erts-5.9.2) [64-bit] [smp:12:12]
>> [async-threads:30]\n"},
>>
>> {memory,
>>
>> [{total,8512001360},
>>
>> {processes,7401925315},
>>
>> {processes_used,7401754972},
>>
>> {system,1110076045},
>>
>> {atom,662409},
>>
>> {atom_used,652416},
>>
>> {binary,857726336},
>>
>> {code,17969722},
>>
>> {ets,4550280}]},
>>
>> {vm_memory_high_watermark,0.4},
>>
>> {vm_memory_limit,10304190873},
>>
>> {disk_free_limit,1000000000},
>>
>> {disk_free,39365828608},
>>
>> {file_descriptors,
>>
>> [{total_limit,16284},
>>
>> {total_used,142},
>>
>> {sockets_limit,14653},
>>
>> {sockets_used,113}]},
>>
>> {processes,[{limit,1048576},{used,5986}]},
>>
>> {run_queue,0},
>>
>> {uptime,104350}]
>>
>> ...done.
>>
>> Best regards,
>>
>> --Bin Chen
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>
--
Simon MacMullen
RabbitMQ, VMware
More information about the rabbitmq-discuss
mailing list