[rabbitmq-discuss] Big backlogs
emile at rabbitmq.com
Mon Oct 31 12:31:31 GMT 2011
On 29/10/11 03:04, Adam Rabung wrote:
> I have come to understand that a happy Rabbit is a nearly-empty Rabbit.
> Obviously, no matter how you lay things out, the publish rate will
> outpace the consumption rate, sometimes for sustained periods of time.
If that was always true then it would not be possible for a queue to
ever get shorter. In the general case it should be possible for
consumers to catch up with producers.
> Another approach would be to re-route backlogs to a "throttling
> consumer". This consumer quickly drains the queue, storing messages to
> a durable store. At a prescribed rate, it pulls these messages out in
> FIFO order and feeds them back in to the general queue. In general, it
> seems useful in many cases to move time-insensitive messages aside
> temporarily when resources are tight.
Rabbit internally allocates RAM in a way that favours fast moving
queues, and is more likely to place messages from slow moving queues on
disk. Your description sounds somewhat similar.
> My question is: would another RabbitMQ broker be appropriate "durable
> store" for storing these throttled messages? We can assume there will
> be far too many message to store in memory. I realize Rabbit will page
> large queues off to Mnesia, but will it perform reasonably if only 10%
> of messages can be stored in memory? 1%? While this is a FIFO
> structure, it seems to have many characteristics that are different from
> a typical queue: very large footprint, very low throughput, very few
If you have distinct streams of messages that have different
requirements and characteristics then you should consider using
different brokers. From your description some messages are
time-insensitive. If you want to provide better guarantees that
time-sensitive messages are processed quickly then you should consider a
dedicated broker with adequate RAM.
More information about the rabbitmq-discuss