[rabbitmq-discuss] RabbitMQ with many queues

[Emile Joubert]

Hi Eric,

On 15/04/13 22:31, Eric Berg wrote:
> I am curious about those of you who have run Rabbit clusters with
> many thousands of queues

I have only used that many queues under testing conditions, rather than
production.

This is more relevant if you are using non-HA queues, but can be
relevant even for HA queues that are not mirrored to all nodes: make
sure you achieve a balanced load by not defining all queues on a single
node.

Make sure you provision adequate resources. It is possible to declare an
enormous number of idle queues without difficulty, but the broker will
grind to a halt if many queues on an under-resourced node start
consuming CPU and RAM. Use realistic workloads to estimate the required
resources.

Use a sensible naming scheme for queues so that you can easily manage a
subset. Consider binding queues via internal exchanges if that makes the
management of bindings easier.

Don't perform "rabbitmqctl list_queues" in a tight loop for monitoring
as this will generate significant load and slow down the broker.



Please report back anything you learn from your experience that other
may benefit from.


-Emile