[rabbitmq-discuss] Almost all statistics drop to 0 every hour

[Simon MacMullen]

On 13/11/12 10:35, Irmo Manie wrote:
> I'm still seeing this behavior and it seems to get worse.
> Every hour there is a drop in delivery rate to 0 and now also the
> publish rate drops to 0.
> Sometimes even for a period of 5 minutes
>
> I'm really sure that messaging continues and the statistics are a lie.
> We use rabbitmq as our main communication hub and certainly a drop in
> real-time market data going from ~500 msg/s to 0 for about 5 minutes
> must be noticed somewhere if this is true.
>
> I'm wondering if it might be the writing part of the statistics to the
> database. Could that timeout?

No.

But if the database is busy with something else then it could take a 
while to process the updates it's received. I find it hard to believe 
that it could be busy for 5 minutes though.

There is nothing scheduled once an hour in RabbitMQ though - so whatever 
is happening once an hour must be outside the broker. Figuring out what 
that is would help a lot.

> I don't really want to restart to see if that fixes the problem.
> It's been running for 177 days now and apart from the weird statistics
> we don't have any problems.
>
> Is there anything I could do as additional checks to figure out where
> the problem might be?

Just to be clear: is this just the rates in /api/overview or the ones in 
/api/queues as well?

I'd also be interested to see the output of:

rabbitmqctl eval 'process_info(global:whereis_name(rabbit_mgmt_db), 
[message_queue_len,status, current_stacktrace]).'

when the statistics drop to 0.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware