[rabbitmq-discuss] Fwd: RabbitMQ and queue message limits?

Tue Aug 31 23:54:01 BST 2010

Our group has wanted this sort of functionality also.  The problem we want
to solve
is pretty much the same:  If a queue comes to contain more than X entries,
then it
means that there is a problem with the consumer(s), and we would like to be
notified.

- Jim

Jim Irrer     irrer at umich.edu       (734) 647-4409
University of Michigan Hospital Radiation Oncology
519 W. William St.             Ann Arbor, MI 48103

---------- Forwarded message ----------
From: Matthias Radestock <matthias at rabbitmq.com>
Date: Tue, Aug 31, 2010 at 5:29 PM
Subject: Re: [rabbitmq-discuss] RabbitMQ and queue message limits?
To: Christian Legnitto <clegnitto at mozilla.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com

Christian,

Christian Legnitto wrote:

> My main issue is I intend to have hundreds of queues with 100k+ messages
> that may or may not be processed in a timely manner. Memory is ok with the
> new persister (yay!), but the disk fills up quickly. Right now with only 1
> consumer, if I don't ack messages coming from bugzilla for 2 days I use up
> 20GB of disk. I can of course throw hardware at the problem, but it seems
> I'd reach a point where it merely wouldn't scale.
>

Thanks for explaining the problem you are trying to solve.

Note that in RabbitMQ messages are shared between queues, i.e. if the same
message ends up in multiple queues we have at most one copy in memory and at
most one copy on disk.

So you may find RabbitMQ scales better here than you expected.

 I don't think it is unreasonable to tell consumers that all messages
> up to a point will be persisted, but once a threshold is reached only
> x amount of messages will persist. I see other brokers support such a
> thing, though it sounds like RabbitMQ likely does not.
>

We did introduce queue leases in the 2.0.0 release, which allows the server
to remove entire queues when they haven't been used for a while. That may
not be the best fit for your use case, but it's certainly a feature aimed at
addressing similar problems.

Another feature we have in our plans is per-queue message TTLs, i.e.
messages will get removed from queues if they are older than a specified
threshold. It sounds like that would fit your use case nicely, and arguably
better than a count-based limit. Though the latter too is something we may
support one day.

Regards,

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100831/9061c2db/attachment.htm>