[rabbitmq-discuss] Reliable way to get number of messages in a queue

[Matthew Sackman]

Hi Michael,

On Thu, Nov 19, 2009 at 04:08:52PM +0000, Michael Nacos wrote:
> I have changed our polling strategy to read the message count from
> queue.declare, it makes more sense, obviously. I am using the Erlang client
> for the polling process. Previous answers such as 'not supported by the
> protocol' had put me in a 'dirty'-solution mindset, which explains the
> initial polling strategy (I used the first thing I had seen which gave me
> that sort of information).

That's fair enough. The erlang client has got a lot better in recent
months and is continuing to improve. It is our next target to move into
a fully supported client mode, but there's still some work to do to get
it to that stage.

> The problem, as you have guessed already, is the number we really want is
> messages_unacknowledged. The number I get from the message count is
> messages_ready. I have been trying to make a producer process self-regulate
> its firing rate in case the consumers are not keeping up (to avoid ram
> exhaustion until the new persister comes along -- sorry Matthew :-)

Heh, that's fine - the persister is entering QA and code review stages
now so progress is being made, but as ever, no ETA!

Right, I would suggest that what you do is in the consumers, set QoS to
1. That way consumers won't get flooded with messages and will only get
new messages when they ack the previous one. As a result, if the
messages_ready count on the queue > 0, then the number of consumers is
equal to the number of unacked messages.

> Our Java consumers re-use their channels and consume messages 'by
> subscription' (it's obviously a better approach) without disconnecting at
> any point while the system is running. When even a single consumer connects
> to the queue in question (it's a multiple consumers for a single queue
> round-robin scenario) in this mode, all 'ready' messages become
> 'unacknowledged' messages, even though there is no way our Java threads have
> processed them this fast (do the messages go into an internal Java client
> buffer perhaps? or is this more of a responsibility transfer case?

Yeah, without QoS set, they all will get sent down to consumers who will
buffer them.

> ). So our
> polling process constantly gets zero (messages_ready), unless all consumers
> disconnect, in which case messages_ready shoots up to whatever
> messages_unacknowledged is when this happens. Is this what you would expect?

Yup. QoS is your friend here ;)

> For our purposes, anything less than an ack means our consumers are not
> keeping up, so we have enforced a qos setting of 1 for each consumer. This
> way, the message_ready value returned to the polling process will, of
> course, be off by a few messages each time but, at least, its order of
> magnitude indicates if our consumers are keeping up.

Yes, that's the right thing to do. Some other users do something very
similar - set QoS to 1 and then if the messages_ready count is non zero,
start up a new consumer process, thus allowing you to grow your
parallelism on the consuming side dynamically with load, which is fairly
neat. I'm not sure whether you're able to do that - I get the impression
that you have a static number of consumers and are trying to throttle at
the producers.

One other idea you could play with is to set the
system_memory_high_watermark configuration value (see the bottom of
http://www.rabbitmq.com/extensions.html) to a very low value, and then
as soon as there's any buildup in Rabbit, it'll raise the channel.flow
flag to the producers, thus stopping them from sending anything in. This
may be sufficient, but it depends somewhat on your platform, resources
etc. Might be worth investigating though.

Matthew