[rabbitmq-discuss] RabbitMQ and queue message limits?

[Christian Legnitto]

Comments inline.

On Aug 31, 2010, at 2:29 PM, Matthias Radestock wrote:

> Christian,
> 
> Christian Legnitto wrote:
>> My main issue is I intend to have hundreds of queues with 100k+ messages that may or may not be processed in a timely manner. Memory is ok with the new persister (yay!), but the disk fills up quickly. Right now with only 1 consumer, if I don't ack messages coming from bugzilla for 2 days I use up 20GB of disk. I can of course throw hardware at the problem, but it seems I'd reach a point where it merely wouldn't scale.
> 
> Thanks for explaining the problem you are trying to solve.

No problem, I know my use-case isn't the common case, but I think it is fairly common as far as message brokers at large orgs go.

> 
> Note that in RabbitMQ messages are shared between queues, i.e. if the same message ends up in multiple queues we have at most one copy in memory and at most one copy on disk.
> 
> So you may find RabbitMQ scales better here than you expected.

That's good to know, though I imagine there will be scaling issues when I expand to different sorts of high-volume messages. Plus, at 20 GB every couple of days (before instituting all message types) I will likely need to find some sort of long-term solution.

> 
>> I don't think it is unreasonable to tell consumers that all messages
>> up to a point will be persisted, but once a threshold is reached only
>> x amount of messages will persist. I see other brokers support such a
>> thing, though it sounds like RabbitMQ likely does not.
> 
> We did introduce queue leases in the 2.0.0 release, which allows the server to remove entire queues when they haven't been used for a while. That may not be the best fit for your use case, but it's certainly a feature aimed at addressing similar problems.

Yep, it might work though would be less-than ideal. I'll look into it, though I would hate to clear a queue with 10 messages sitting in it because the consumer was down for a month or so.

> Another feature we have in our plans is per-queue message TTLs, i.e. messages will get removed from queues if they are older than a specified threshold. It sounds like that would fit your use case nicely, and arguably better than a count-based limit. Though the latter too is something we may support one day.

Again, I'm not sure this really covers my case. For my use case I only care about the timeliness of the message as long as I am resource constrained. If I have ample resources I'd be willing to have the messages sit there for months.

Thanks for the prompt reply and I'll look into the queue leases.

Christian