[rabbitmq-discuss] Reliable way to get number of messages in a queue

[Michael Nacos]

Thanks again for your help.

I have changed our polling strategy to read the message count from
queue.declare, it makes more sense, obviously. I am using the Erlang client
for the polling process. Previous answers such as 'not supported by the
protocol' had put me in a 'dirty'-solution mindset, which explains the
initial polling strategy (I used the first thing I had seen which gave me
that sort of information).

The problem, as you have guessed already, is the number we really want is
messages_unacknowledged. The number I get from the message count is
messages_ready. I have been trying to make a producer process self-regulate
its firing rate in case the consumers are not keeping up (to avoid ram
exhaustion until the new persister comes along -- sorry Matthew :-)

Our Java consumers re-use their channels and consume messages 'by
subscription' (it's obviously a better approach) without disconnecting at
any point while the system is running. When even a single consumer connects
to the queue in question (it's a multiple consumers for a single queue
round-robin scenario) in this mode, all 'ready' messages become
'unacknowledged' messages, even though there is no way our Java threads have
processed them this fast (do the messages go into an internal Java client
buffer perhaps? or is this more of a responsibility transfer case?). So our
polling process constantly gets zero (messages_ready), unless all consumers
disconnect, in which case messages_ready shoots up to whatever
messages_unacknowledged is when this happens. Is this what you would expect?

For our purposes, anything less than an ack means our consumers are not
keeping up, so we have enforced a qos setting of 1 for each consumer. This
way, the message_ready value returned to the polling process will, of
course, be off by a few messages each time but, at least, its order of
magnitude indicates if our consumers are keeping up.

Best regards,

Michael

2009/11/18 Michael Nacos <m.nacos at gmail.com>

> thanks... more testing tomorrow
>
> 2009/11/18 Matthew Sackman <matthew at lshift.net>
>
> On Wed, Nov 18, 2009 at 05:18:53PM +0000, Matthew Sackman wrote:
>> > That's odd. However, you're really much much better off using
>> > queue.declare to redeclare the existing queue - the declare-ok that you
>> > get back contains the number of messages in the queue. This is a much
>> > neater strategy than below. I would recommend you switch to using this
>> > and repeat your test.
>>
>> What's actually wrong with what you were doing is that the rabbitmqctl
>> list_queues contains both the ready messages and the unack'd msgs (as
>> well as uncommitted msgs). Thus if your consumers aren't acking msgs
>> quickly enough (and have an unbounded qos setting) then it will appear
>> that they're not keeping up. Use:
>>
>> rabbitmqctl list_queues name messages_ready messages_unacknowledged
>>
>> Matthew
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20091119/c2b492f1/attachment.htm