[rabbitmq-discuss] Determining age of queue contents

[Simon MacMullen]

On 22/01/12 14:23, Alexander Staubo wrote:
> [Please CC as I am not on the list.]
>
> The fine-grained stats plugin provides information about message
> processing rate and queue lengths, which can be used to infer whether
> a queue is "lagging". Two problems:
>
> 1. The only way to reliably infer the true processing rate of a queue
> is to implement a stateful watching algorithm, being a function both
> of the current processing rate as well as the *growth rate* of the
> queue; a queue that grows faster than messages are processes is a
> potential bottleneck. While fairly trivial to write, when you merely
> want to monitor queues from a Nagios plugin, I would wish this was
> something built into RabbitMQ itself, closest to where the
> information is stored. A Nagios plugin would have to store state data
> in the file system.

Why would such a plugin need to be stateful? Note that growth rate (only 
since the last sample, like all the rates we show) is available from 
mgmt as of 2.7.1. Was that what you meant? If not can you explain what 
calculation you would want?

> 2. Even when looking at processing rate and growth rate, there is not
> enough sufficient information to determine a realistic churn rate.
> What's really needed is to track the age of the newest and last
> messages in the queue. If you have this age difference, you can
> define acceptables deadline for message processing in a monitoring
> system, which is a better heuristic than the processing rate.
>
> Would it be possible to implement a way to extract a per-queue age
> value (in a performant way)?

This could potentially happen, but it'd be quite intrusive so there are 
other things higher up the priority list right now...

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware