[rabbitmq-discuss] RabbitMQ and queue message limits?

Tue Aug 31 22:16:11 BST 2010

My main issue is I intend to have hundreds of queues with 100k+ messages that may or may not be processed in a timely manner. Memory is ok with the new persister (yay!), but the disk fills up quickly. Right now with only 1 consumer, if I don't ack messages coming from bugzilla for 2 days I use up 20GB of disk. I can of course throw hardware at the problem, but it seems I'd reach a point where it merely wouldn't scale. I don't think it is unreasonable to tell consumers that all messages up to a point will be persisted, but once a threshold is reached only x amount of messages will persist. I see other brokers support such a thing, though it sounds like RabbitMQ likely does not.

Thanks,
Christian

On Aug 31, 2010, at 1:50 PM, Matthias Radestock wrote:

> Christian,
> 
> Christian Legnitto wrote:
>> Does RabbitMQ currently support (or are there plans to) something
>> like:
>> http://activemq.apache.org/slow-consumer-handling.html
>> I'd like to define a max size for a queue and configure the behavior
>> when the threshold is released. I guess I could have some cleanup
>> consumer that drains the queues periodically, but that seems
>> error-prone and brittle.
> 
> What's the problem with just keeping the messages in the queue? RabbitMQ automatically pages messages to disk, so it's not bounded by memory. And the disk paging is quite clever in that it gives more memory to fast moving queues.
> 
> 
> Regards,
> 
> Matthias.