[rabbitmq-discuss] Big backlogs

[Adam Rabung]

Hello,
I have come to understand that a happy Rabbit is a nearly-empty Rabbit.
 Obviously, no matter how you lay things out, the publish rate will outpace
the consumption rate, sometimes for sustained periods of time.  My first
reaction is to throttle producers, but due to the number and behavior of
producers in my system, this seems difficult.

Another approach would be to re-route backlogs to a "throttling consumer".
 This consumer quickly drains the queue, storing messages to a durable
store.  At a prescribed rate, it pulls these messages out in FIFO order and
feeds them back in to the general queue.  In general, it seems useful in
many cases to move time-insensitive messages aside temporarily when
resources are tight.

My question is: would another RabbitMQ broker be appropriate "durable store"
for storing these throttled messages?  We can assume there will be far too
many message to store in memory.  I realize Rabbit will page large queues
off to Mnesia, but will it perform reasonably if only 10% of messages can be
stored in memory?  1%?  While this is a FIFO structure, it seems to have
many characteristics that are different from a typical queue: very large
footprint, very low throughput, very few consumers.

Thank you for any feedback.
Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111028/93791edb/attachment.htm>