[rabbitmq-discuss] RabbitMQ with many queues

[Eric Berg]

Thanks for the reply Emile. Right now we have a 4 node cluster with primary
and failover haproxy round robin load balancers, so dynamic generation of
queues tend to be pretty well distributed especially with a 2 node ha
policy. I guess it will just take some close monitoring to verify that the
resources are not being exhausted, though even adding new nodes to the
cluster would not help in that case correct? Since in a 4 node cluster with
a 2 node ha policy, only newly created queues would be replicated to the
new node? How do others handle situations like this?

- PHP -
On a related note, we have had some unfortunate experiences when testing a
few different php libraries and Rabbit kicks in its flow control. It looks
like that our producers using various php libs in an Apache setting dont
handle the flow control well, while the php connection dies Rabbit still
thinks they are active even though no producing is coming across, filling
up the file descriptors.

Thanks guys!




On Wed, Apr 17, 2013 at 11:23 AM, Emile Joubert <emile at rabbitmq.com> wrote:

>
> Hi Eric,
>
> On 15/04/13 22:31, Eric Berg wrote:
> > I am curious about those of you who have run Rabbit clusters with
> > many thousands of queues
>
> I have only used that many queues under testing conditions, rather than
> production.
>
> This is more relevant if you are using non-HA queues, but can be
> relevant even for HA queues that are not mirrored to all nodes: make
> sure you achieve a balanced load by not defining all queues on a single
> node.
>
> Make sure you provision adequate resources. It is possible to declare an
> enormous number of idle queues without difficulty, but the broker will
> grind to a halt if many queues on an under-resourced node start
> consuming CPU and RAM. Use realistic workloads to estimate the required
> resources.
>
> Use a sensible naming scheme for queues so that you can easily manage a
> subset. Consider binding queues via internal exchanges if that makes the
> management of bindings easier.
>
> Don't perform "rabbitmqctl list_queues" in a tight loop for monitoring
> as this will generate significant load and slow down the broker.
>
>
>
> Please report back anything you learn from your experience that other
> may benefit from.
>
>
> -Emile
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130418/87c3cdea/attachment.htm>