[rabbitmq-discuss] RabbitMQ and queue message limits?

Thu Sep 2 15:23:06 BST 2010

Hi Christian,

Christian Legnitto <clegnitto at mozilla.com> writes:
> My main issue is I intend to have hundreds of queues with 100k+
> messages that may or may not be processed in a timely manner. Memory
> is ok with the new persister (yay!), but the disk fills up
> quickly. Right now with only 1 consumer, if I don't ack messages
> coming from bugzilla for 2 days I use up 20GB of disk. I can of course
> throw hardware at the problem, but it seems I'd reach a point where it
> merely wouldn't scale. I don't think it is unreasonable to tell
> consumers that all messages up to a point will be persisted, but once
> a threshold is reached only x amount of messages will persist. I see
> other brokers support such a thing, though it sounds like RabbitMQ
> likely does not.

It should be possible to achieve what you describe outside of the
RabbitMQ broker: A program could periodically use rabbitmqctl (or
whatever new improved monitoring interfaces we might add) to find queues
that are over their limit. And then, where necessary, it could consume
from queues to reduce their size.

You may already realize that, but it seems worth pointing out for the
general rabbitmq-discuss audience.

David

-- 
David Wragg
Staff Engineer, RabbitMQ
SpringSource, a division of VMware