[rabbitmq-discuss] RabbitMQ and queue message limits?

[Matthias Radestock]

Christian,

Christian Legnitto wrote:
> My main issue is I intend to have hundreds of queues with 100k+ 
> messages that may or may not be processed in a timely manner. Memory 
> is ok with the new persister (yay!), but the disk fills up quickly. 
> Right now with only 1 consumer, if I don't ack messages coming from 
> bugzilla for 2 days I use up 20GB of disk. I can of course throw 
> hardware at the problem, but it seems I'd reach a point where it 
> merely wouldn't scale.

Thanks for explaining the problem you are trying to solve.

Note that in RabbitMQ messages are shared between queues, i.e. if the 
same message ends up in multiple queues we have at most one copy in 
memory and at most one copy on disk.

So you may find RabbitMQ scales better here than you expected.

> I don't think it is unreasonable to tell consumers that all messages
> up to a point will be persisted, but once a threshold is reached only
> x amount of messages will persist. I see other brokers support such a
> thing, though it sounds like RabbitMQ likely does not.

We did introduce queue leases in the 2.0.0 release, which allows the 
server to remove entire queues when they haven't been used for a while. 
That may not be the best fit for your use case, but it's certainly a 
feature aimed at addressing similar problems.

Another feature we have in our plans is per-queue message TTLs, i.e. 
messages will get removed from queues if they are older than a specified 
threshold. It sounds like that would fit your use case nicely, and 
arguably better than a count-based limit. Though the latter too is 
something we may support one day.


Regards,

Matthias.