[rabbitmq-discuss] RabbitMQ and queue message limits?

Tue Aug 31 22:24:33 BST 2010

Hi!

Not to hijack anyones thread, but I thought I would chime in to second this feature-request :) I've got a couple of queues that might fill the disk if consumers are not working properly, and I still want the rest of the "slow moving" queues to live on.

In my case it would be nice if the server could throw out old messages (one by one as new messages come in) once the queue reaches a storage threshold/message count.

regards,
Øyvind

On Aug 31, 2010, at 11:16 PM, Christian Legnitto wrote:

> My main issue is I intend to have hundreds of queues with 100k+ messages that may or may not be processed in a timely manner. Memory is ok with the new persister (yay!), but the disk fills up quickly. Right now with only 1 consumer, if I don't ack messages coming from bugzilla for 2 days I use up 20GB of disk. I can of course throw hardware at the problem, but it seems I'd reach a point where it merely wouldn't scale. I don't think it is unreasonable to tell consumers that all messages up to a point will be persisted, but once a threshold is reached only x amount of messages will persist. I see other brokers support such a thing, though it sounds like RabbitMQ likely does not.
> 
> Thanks,
> Christian
> 
> On Aug 31, 2010, at 1:50 PM, Matthias Radestock wrote:
> 
>> Christian,
>> 
>> Christian Legnitto wrote:
>>> Does RabbitMQ currently support (or are there plans to) something
>>> like:
>>> http://activemq.apache.org/slow-consumer-handling.html
>>> I'd like to define a max size for a queue and configure the behavior
>>> when the threshold is released. I guess I could have some cleanup
>>> consumer that drains the queues periodically, but that seems
>>> error-prone and brittle.
>> 
>> What's the problem with just keeping the messages in the queue? RabbitMQ automatically pages messages to disk, so it's not bounded by memory. And the disk paging is quite clever in that it gives more memory to fast moving queues.
>> 
>> 
>> Regards,
>> 
>> Matthias.
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss