I'm still seeing this behavior and it seems to get worse. <div>Every hour there is a drop in delivery rate to 0 and now also the publish rate drops to 0.</div><div>Sometimes even for a period of 5 minutes<br><div><br></div>
<div>I'm really sure that messaging continues and the statistics are a lie.</div><div>We use rabbitmq as our main communication hub and certainly a drop in real-time market data going from ~500 msg/s to 0 for about 5 minutes must be noticed somewhere if this is true.</div>
<div><br></div><div>I'm wondering if it might be the writing part of the statistics to the database. Could that timeout?</div><div><br></div><div>I don't really want to restart to see if that fixes the problem. </div>
<div>It's been running for 177 days now and apart from the weird statistics we don't have any problems.</div><div><br></div><div>Is there anything I could do as additional checks to figure out where the problem might be?</div>
<div><br></div><div>- Irmo<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Nov 6, 2012 at 12:24 PM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 06/11/12 08:48, Irmo Manie wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
As you can see, the delivery (no-ack) rate drops to 0 every hour for a<br>
couple of minutes. Other statistics on message processing indicate that<br>
actual delivery/consuming is continuing, so it looks like it's really<br>
only the statistics that are wrong.<br>
<br>
If I look at the json output during such a 'downtime' I can see that the<br>
publish rate is always available, but the rest of the rates<br>
(acknowledge, delivery, etc) are mostly 0.<br>
</blockquote>
<br></div>
Just the rates in /api/overview or the ones in /api/queues as well?<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I understand a bit from the sources that in this /api/overview call from<br>
the management api, these statistics are gathered from different places,<br>
both memory and database (with additional calculations executed on them<br>
while fetching), correct?<br>
</blockquote>
<br></div>
Yes. We get basic queue information from the Mnesia database, and then augment it with statistics information (including rates) from the (in-memory) management database.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Could it be that some of these values are 0 because of certain time-outs<br>
while getting the data?<br>
</blockquote>
<br></div>
No, the requests for stats don't time out.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In other words: do I have to start searching for<br>
the problem at the database disk/IO level?<br>
</blockquote>
<br></div>
No.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I couldn't see any IO waits on the machine indicating something else is<br>
happening at the time. Also CPU load is normal.<br>
Still the weird thing is that this also happens on a test machine at<br>
roughly the same times.<br>
Since both machines are VM's, this might indicate that it could be an<br>
infrastructural problem, but I'd like to be sure before accusing 'others'.<br>
</blockquote>
<br></div>
If /api/overview and /api/queues contradict each other I'd like to know about it.<br>
<br>
But if the rates drop to 0 in /api/queues as well, Occam's Razor suggests that maybe your consumers are pausing for some reason :)<br>
<br>
Cheers, Simon<span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</font></span></blockquote></div><br></div></div></div>