Our group has wanted this sort of functionality also. The problem we want to solve<br>is pretty much the same: If a queue comes to contain more than X entries, then it<br>means that there is a problem with the consumer(s), and we would like to be notified.<br>
<br>- Jim<br><br>Jim Irrer <a href="mailto:irrer@umich.edu">irrer@umich.edu</a> (734) 647-4409<br>University of Michigan Hospital Radiation Oncology<br>519 W. William St. Ann Arbor, MI 48103<br>
<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Matthias Radestock</b> <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com">matthias@rabbitmq.com</a>></span><br>
Date: Tue, Aug 31, 2010 at 5:29 PM<br>Subject: Re: [rabbitmq-discuss] RabbitMQ and queue message limits?<br>To: Christian Legnitto <<a href="mailto:clegnitto@mozilla.com">clegnitto@mozilla.com</a>><br>Cc: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<br><br>Christian,<div class="im"><br>
<br>
Christian Legnitto wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
My main issue is I intend to have hundreds of queues with 100k+ messages that may or may not be processed in a timely manner. Memory is ok with the new persister (yay!), but the disk fills up quickly. Right now with only 1 consumer, if I don't ack messages coming from bugzilla for 2 days I use up 20GB of disk. I can of course throw hardware at the problem, but it seems I'd reach a point where it merely wouldn't scale.<br>
</blockquote>
<br></div>
Thanks for explaining the problem you are trying to solve.<br>
<br>
Note that in RabbitMQ messages are shared between queues, i.e. if the same message ends up in multiple queues we have at most one copy in memory and at most one copy on disk.<br>
<br>
So you may find RabbitMQ scales better here than you expected.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
I don't think it is unreasonable to tell consumers that all messages<br>
up to a point will be persisted, but once a threshold is reached only<br>
x amount of messages will persist. I see other brokers support such a<br>
thing, though it sounds like RabbitMQ likely does not.<br>
</blockquote>
<br></div>
We did introduce queue leases in the 2.0.0 release, which allows the server to remove entire queues when they haven't been used for a while. That may not be the best fit for your use case, but it's certainly a feature aimed at addressing similar problems.<br>
<br>
Another feature we have in our plans is per-queue message TTLs, i.e. messages will get removed from queues if they are older than a specified threshold. It sounds like that would fit your use case nicely, and arguably better than a count-based limit. Though the latter too is something we may support one day.<div>
<div></div><div class="h5"><br>
<br>
<br>
Regards,<br>
<br>
Matthias.<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></div><br>