[rabbitmq-discuss] Bound Queues

[Garrett Smith]

On Tue, Apr 13, 2010 at 12:25 AM, Matthias Radestock
<matthias at lshift.net> wrote:
> Dan,
>
> Tony Garnock-Jones wrote:
>> Dan Di Spaltro wrote:
>>> Is there anyway you can specify a memory/disk/length limit to a queue
>>> size with the action being a similar to a fixed window? [...]
>>
>> It's not exactly on the roadmap, but it's something we've considered before.
>> There are some questions that need careful thought though [...]
>
> One particular tricky issue is sharing. AMQP message may get delivered
> to 0, 1 or *several* queues. RabbitMQ generally only keeps *one* copy of
> a message, regardless of how many queues it ends up in. So queues end up
> sharing messages. And some data can even be shared *between* messages.
>
> Hence there really is no such thing as a per-queue memory/disk footprint.
>
> Furthermore, there is no way to determine the memory footprint of a
> message precisely due to memory fragmentation, gc, etc.

At the end of the day though, when messages pile up in a queue (just
takes one), you'll run out of memory at some point.

Without support for ttl/expires, Rabbit is particularly susceptible to
this "problem". I realize this it's by design and there are cases
where you *want* this, but there are absolutely cases when you
*don't*. It just takes one flaky consumer to put your entire system in
jeopardy.

For this reason, I've moved all of our queues to exclusive + auto
delete. I'd prefer to maintain queue state as producers and consumers
drop off, but the benefit of this is not enough to justify the
risk/cost.

I also have an external monitor on queue size (I believe measured in
memory as per rabbitmqctl output) in the event a rogue consumer
declares a queue with the wrong settings.

FWIW, the qpid server does respect the ttl message property and this
condition is easier to manage as a result. I'm a fan of ttl as it's
application specific (producers get to chose how long their messages
should live) and provides a reasonable heuristic for deleting messages
that aren't consumed in a timely manner.

This has been brought up a few times and I understand that there is a
similar set of questions that need to be addressed. Nonetheless, from
my point of view, this is a topic (general problem) that should get a
bump in road map priority.

Garrett