Here goes, randomly executed the command. Last 5 calls are done when publish rate dropped to 0.<div><br></div><div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,4},</div><div>{status,runnable},</div><div>{current_stacktrace,[{rabbit_mgmt_db,lookup_element,3,[]},</div><div> {rabbit_mgmt_db,handle_stats,6,[]},</div><div> {rabbit_mgmt_db,handle_event,2,[]},</div>
<div> {rabbit_mgmt_db,handle_cast,2,[]},</div><div> {gen_server,handle_msg,5,</div><div> [{file,"gen_server.erl"},{line,597}]},</div><div>
{proc_lib,init_p_do_apply,3,</div><div> [{file,"proc_lib.erl"},{line,227}]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,1887},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,1040},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,3165},</div><div>{status,runnable},</div><div>{current_stacktrace,[{dict,update_bkt,4,[{file,"dict.erl"},{line,321}]},</div><div> {dict,update_bkt,4,[{file,"dict.erl"},{line,324}]},</div>
<div> {dict,update_bkt,4,[{file,"dict.erl"},{line,324}]},</div><div> {dict,on_bucket,3,[{file,"dict.erl"},{line,414}]},</div><div> {dict,update,4,[{file,"dict.erl"},{line,317}]},</div>
<div> {lists,foldl,3,[{file,"lists.erl"},{line,1197}]},</div><div> {rabbit_mgmt_db,group_sum,2,[]},</div><div> {rabbit_mgmt_db,'-fine_stats_fun/2-lc$^0/1-0-',2,[]}]}]</div>
<div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div><div>[{message_queue_len,2423},</div><div>{status,runnable},</div>
<div>{current_stacktrace,[{lists,suffix,2,[{file,"lists.erl"},{line,151}]},</div><div> {rabbit_mgmt_db,maybe_zero_rate,2,[]},</div><div> {rabbit_mgmt_db,'-zero_old_rates/2-lc$^0/1-0-',2,[]},</div>
<div> {rabbit_mgmt_db,'-get_fine_stats_from_list/3-lc$^0/1-0-',</div><div> 2,[]},</div><div> {rabbit_mgmt_db,'-get_fine_stats_from_list/3-lc$^0/1-0-',</div>
<div> 2,[]},</div><div> {rabbit_mgmt_db,get_fine_stats_from_list,3,[]},</div><div> {rabbit_mgmt_db,'-fine_stats_fun/2-lc$^0/1-0-',2,[]},</div>
<div> {rabbit_mgmt_db,'-fine_stats_fun/2-lc$^0/1-0-',2,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,1454},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,0},</div><div>{status,waiting},</div><div>{current_stacktrace,[{gen_server,loop,6,[{file,"gen_server.erl"},{line,348}]},</div><div> {proc_lib,init_p_do_apply,3,</div>
<div> [{file,"proc_lib.erl"},{line,227}]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,757},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,876},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,1677},</div><div>{status,runnable},</div><div>{current_stacktrace,[{rabbit_mgmt_db,if_unknown,2,[]},</div><div> {rabbit_mgmt_db,gs_update_add,3,[]},</div><div> {rabbit_mgmt_db,'-gs_update/3-lc$^0/1-0-',3,[]},</div>
<div> {rabbit_mgmt_db,'-gs_update/3-lc$^0/1-0-',3,[]},</div><div> {lists,foldl,3,[{file,"lists.erl"},{line,1197}]},</div><div> {dict,map_bucket,2,[{file,"dict.erl"},{line,459}]},</div>
<div> {dict,map_bucket,2,[{file,"dict.erl"},{line,459}]},</div><div> {dict,map_bkt_list,2,[{file,"dict.erl"},{line,455}]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,0},</div><div>{status,waiting},</div><div>{current_stacktrace,[{gen_server,loop,6,[{file,"gen_server.erl"},{line,348}]},</div><div> {proc_lib,init_p_do_apply,3,</div>
<div> [{file,"proc_lib.erl"},{line,227}]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,735},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,157},</div><div>{status,runnable},</div><div>{current_stacktrace,[{lists,nthtail,2,[{file,"lists.erl"},{line,127}]},</div><div> {lists,suffix,2,[{file,"lists.erl"},{line,153}]},</div>
<div> {rabbit_mgmt_db,maybe_zero_rate,2,[]},</div><div> {rabbit_mgmt_db,'-zero_old_rates/2-lc$^0/1-0-',2,[]},</div><div> {rabbit_mgmt_db,'-zero_old_rates/2-lc$^0/1-0-',2,[]},</div>
<div> {rabbit_mgmt_db,'-get_fine_stats_from_list/3-lc$^0/1-0-',</div><div> 2,[]},</div><div> {rabbit_mgmt_db,'-get_fine_stats_from_list/3-lc$^0/1-0-',</div>
<div> 2,[]},</div><div> {rabbit_mgmt_db,get_fine_stats_from_list,3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,0},</div><div>{status,waiting},</div><div>{current_stacktrace,[{gen_server,loop,6,[{file,"gen_server.erl"},{line,348}]},</div><div> {proc_lib,init_p_do_apply,3,</div>
<div> [{file,"proc_lib.erl"},{line,227}]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,3419},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,3942},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,1940},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,0},</div><div>{status,waiting},</div><div>{current_stacktrace,[{gen_server,loop,6,[{file,"gen_server.erl"},{line,348}]},</div><div> {proc_lib,init_p_do_apply,3,</div>
<div> [{file,"proc_lib.erl"},{line,227}]}]}]</div><div>...done.</div><div>chvirrmqprd01:~ # rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'</div>
<div>[{message_queue_len,1102},</div><div>{status,runnable},</div><div>{current_stacktrace,[{mnesia_lib,db_fixtable,3,</div><div> [{file,"mnesia_lib.erl"},{line,1127}]},</div><div>
{mnesia_lib,db_match_object,3,</div><div> [{file,"mnesia_lib.erl"},{line,1087}]},</div><div> {rpc,local_call,3,[{file,"rpc.erl"},{line,327}]},</div>
<div> {mnesia,do_dirty_rpc,5,</div><div> [{file,"mnesia.erl"},{line,1807}]},</div><div> {rabbit_mgmt_db,augment_queue_pid,2,[]},</div><div> {rabbit_mgmt_db,augment,4,[]},</div>
<div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]},</div><div> {rabbit_mgmt_db,'-augment/3-lc$^0/1-0-',3,[]}]}]</div><div>...done.</div></div><div><br></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Nov 13, 2012 at 3:23 PM, Irmo Manie <span dir="ltr"><<a href="mailto:irmo.manie@gmail.com" target="_blank">irmo.manie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Tue, Nov 13, 2012 at 12:24 PM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br></div><div class="gmail_extra"><div class="gmail_quote">
<div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 13/11/12 10:35, Irmo Manie wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm still seeing this behavior and it seems to get worse.<br>
Every hour there is a drop in delivery rate to 0 and now also the<br>
publish rate drops to 0.<br>
Sometimes even for a period of 5 minutes<br>
<br>
I'm really sure that messaging continues and the statistics are a lie.<br>
We use rabbitmq as our main communication hub and certainly a drop in<br>
real-time market data going from ~500 msg/s to 0 for about 5 minutes<br>
must be noticed somewhere if this is true.<br>
<br>
I'm wondering if it might be the writing part of the statistics to the<br>
database. Could that timeout?<br>
</blockquote>
<br></div>
No.<br>
<br>
But if the database is busy with something else then it could take a while to process the updates it's received. I find it hard to believe that it could be busy for 5 minutes though.<br>
<br>
There is nothing scheduled once an hour in RabbitMQ though - so whatever is happening once an hour must be outside the broker. Figuring out what that is would help a lot.<div><br></div></blockquote><div><br></div>
</div><div>Clear :-)</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I don't really want to restart to see if that fixes the problem.<br>
It's been running for 177 days now and apart from the weird statistics<br>
we don't have any problems.<br>
<br>
Is there anything I could do as additional checks to figure out where<br>
the problem might be?<br>
</blockquote>
<br></div>
Just to be clear: is this just the rates in /api/overview or the ones in /api/queues as well?<br>
<br></blockquote><div><br></div></div><div>Both show 0 as the rates</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'd also be interested to see the output of:<br>
<br>
rabbitmqctl eval 'process_info(global:whereis_<u></u>name(rabbit_mgmt_db), [message_queue_len,status, current_stacktrace]).'<br>
<br>
when the statistics drop to 0.<div><div><br></div></div></blockquote><div><br></div></div><div>I'll try to check this and let you know if I find anything weird</div><div><br></div><div>Thanks so far for the information and help.</div>
<span class="HOEnZb"><font color="#888888">
<div><br></div><div>- Irmo</div></font></span><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
<br>
Cheers, Simon<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</div></div></blockquote></div></div><br></div>
</blockquote></div><br></div>