[rabbitmq-discuss] Active consumers in declare-ok is less than the actual number of consumers

[Gerolf Seitz]

On Tue, Mar 20, 2012 at 12:17 PM, Simon MacMullen <simon at rabbitmq.com>wrote:

> On 20/03/12 06:43, Gerolf Seitz wrote:
>
>> When passively declaring a queue, the returned consumer count is not the
>> same as the consumer count via the management plugin / http API. Turns
>> out, rabbitmq returns the number of consumers that are ready to receive
>> a message.
>>
>> As an example: there are 5 (manually acknowledging) consumers, 3 of
>> which have consumed a message from the queue, but have not sent an ACK
>> for the respective messages to the broker. Declaring a queue now returns
>> a declare-ok with consumer-count == 2, whereas I would have expected it
>> to be 5.
>>
>> Looking at the rabbitmq-server code reveals that the "active consumer"
>> part of the state is used for getting the next consumer for round robin
>> delivery and returning the number of consumers as part of the declare-ok
>> message.
>>
>> Since I don't suspect this to be an oversight but rather a conscious
>> decision, I'm interested in the rationale behind it. I'm aware that
>> changing the semantics at this point is probably not possible and I may
>> have to resort to fetching the consumer count via the http api (which
>> brings some organizational overhead).
>>
>> Can anybody shed some light on this?
>>
>
> Hi Gerolf.
>
> Yes, this is deliberate. Code archaeology yielded this:
>
> http://hg.rabbitmq.com/**rabbitmq-server/rev/**32e357ad03d6<http://hg.rabbitmq.com/rabbitmq-server/rev/32e357ad03d6>
>
> "count queue consumers as required by the spec".
>
> The spec actually says:
>
>    Reports the number of active consumers for the queue. Note that
>    consumers can  suspend activity (Channel.Flow) in which case they
>    do not appear in this count.
>
> So that is slightly ambiguous - we are excluding consumers that have
> issued channel.flow, but also those which are blocked due to qos / ack
> interaction. But those could easily be defined as "inactive"...
>
> Hmm. I'll have a think about this. What are you trying to do?
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>

The reply-to-author setting of rabbitmq-discuss screwed me ;)



Hi,

The use case is this:

We have a pool of consumers (each running in a heavy weight process) that
can only process a single unit of work at a time each.
The messages that are sent to these consumers contain data that can become
"old" quite quickly (think stock ticks), so queuing hundreds of messages is
not an option.
Also, these consumers need to be kept busy all the time: as soon as a unit
of work is done, the next task will be submitted and picked up by the free
consumer. (prefetch=1, autoack=false).

Based on the number of consumers, the system decides which unit of work is
sent next, mostly based on the previous time the unit of work took to
complete.

Example:
7 consumers overall
Per configuration, only up to 4 consumers may be used to process really
long running tasks (eg. >1 minute task duration)
The rest of the consumers is used for short running tasks. (eg. <1 minute)

The reason why we don't have 2 different queues is twofold:
1) if there are no long running tasks to be executed, all consumers should
be used for short running tasks
2) the decision whether a task is short or long running is made by the
system that sends the messages to the broker and the decision process can
be tweaked via configuration. This is easier to manipulate than
reconfiguring and restarting the heavy weight consumer processes.

I hope this clears it up a bit ;)

Cheers


-- 
Gerolf Seitz

twitter: @gersei
code: github.com/gseitz
blog: www.gerolfseitz.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120320/e98dbf24/attachment.htm>