[rabbitmq-discuss] Big backlogs

[Emile Joubert]

Hi Adam,

On 29/10/11 03:04, Adam Rabung wrote:
> Hello,
> I have come to understand that a happy Rabbit is a nearly-empty Rabbit.
>  Obviously, no matter how you lay things out, the publish rate will
> outpace the consumption rate, sometimes for sustained periods of time.

If that was always true then it would not be possible for a queue to
ever get shorter. In the general case it should be possible for
consumers to catch up with producers.

> Another approach would be to re-route backlogs to a "throttling
> consumer".  This consumer quickly drains the queue, storing messages to
> a durable store.  At a prescribed rate, it pulls these messages out in
> FIFO order and feeds them back in to the general queue.  In general, it
> seems useful in many cases to move time-insensitive messages aside
> temporarily when resources are tight.  

Rabbit internally allocates RAM in a way that favours fast moving
queues, and is more likely to place messages from slow moving queues on
disk. Your description sounds somewhat similar.

> My question is: would another RabbitMQ broker be appropriate "durable
> store" for storing these throttled messages?  We can assume there will
> be far too many message to store in memory.  I realize Rabbit will page
> large queues off to Mnesia, but will it perform reasonably if only 10%
> of messages can be stored in memory?  1%?  While this is a FIFO
> structure, it seems to have many characteristics that are different from
> a typical queue: very large footprint, very low throughput, very few
> consumers.

If you have distinct streams of messages that have different
requirements and characteristics then you should consider using
different brokers. From your description some messages are
time-insensitive. If you want to provide better guarantees that
time-sensitive messages are processed quickly then you should consider a
dedicated broker with adequate RAM.


-Emile