[rabbitmq-discuss] basic.qos prefetch across multiple queues - useful?

Mon Mar 18 20:39:29 GMT 2013

On Mar 18, 2013, at 6:30 PM, Matthias Radestock <matthias at rabbitmq.com> wrote:

> Is anybody setting a basic.qos prefetch limit on channels with consumers on multiple queues?
> 
> If so, are you relying on the fact that the prefetch limit applies *across* queues rather than *per* queue?

Celery is relying on it, but only because that's how QoS is defined in the spec.
It does sound like we'd end up with a lot more messages in memory if it was per queue.  Some users like to configure workers so that they will only consume
as many messages as there are available workers in the execution pool, and so I
guess for this to still work we would need another semaphore/barrier if the prefetch count behavior changes.  It can probably be solved either way.

> 
> That feature entails considerable implementation complexity and has a substantial performance impact on consuming from basic.qos-limited channels in general. And I've just found a bug in that area of the code, fixing of which will make the implementation even more complex and hurt performance even more.

Strange, I would've guessed implementing it on a per-queue basis would be more complex

> 
> Hence I am trying to gauge people's opinion on whether we could make the prefetch limit apply per-queue instead.
> 
> Regards,
> 
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss