<div dir="ltr">Thanks for the reply Emile. Right now we have a 4 node cluster with primary and failover haproxy round robin load balancers, so dynamic generation of queues tend to be pretty well distributed especially with a 2 node ha policy. I guess it will just take some close monitoring to verify that the resources are not being exhausted, though even adding new nodes to the cluster would not help in that case correct? Since in a 4 node cluster with a 2 node ha policy, only newly created queues would be replicated to the new node? How do others handle situations like this?<div>
<br><div style>- PHP -</div><div>On a related note, we have had some unfortunate experiences when testing a few different php libraries and Rabbit kicks in its flow control. It looks like that our producers using various php libs in an Apache setting dont handle the flow control well, while the php connection dies Rabbit still thinks they are active even though no producing is coming across, filling up the file descriptors. </div>
<div><br></div><div style>Thanks guys!</div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 17, 2013 at 11:23 AM, Emile Joubert <span dir="ltr"><<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Eric,<br>
<div class="im"><br>
On 15/04/13 22:31, Eric Berg wrote:<br>
> I am curious about those of you who have run Rabbit clusters with<br>
> many thousands of queues<br>
<br>
</div>I have only used that many queues under testing conditions, rather than<br>
production.<br>
<br>
This is more relevant if you are using non-HA queues, but can be<br>
relevant even for HA queues that are not mirrored to all nodes: make<br>
sure you achieve a balanced load by not defining all queues on a single<br>
node.<br>
<br>
Make sure you provision adequate resources. It is possible to declare an<br>
enormous number of idle queues without difficulty, but the broker will<br>
grind to a halt if many queues on an under-resourced node start<br>
consuming CPU and RAM. Use realistic workloads to estimate the required<br>
resources.<br>
<br>
Use a sensible naming scheme for queues so that you can easily manage a<br>
subset. Consider binding queues via internal exchanges if that makes the<br>
management of bindings easier.<br>
<br>
Don't perform "rabbitmqctl list_queues" in a tight loop for monitoring<br>
as this will generate significant load and slow down the broker.<br>
<br>
<br>
<br>
Please report back anything you learn from your experience that other<br>
may benefit from.<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
-Emile<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</font></span></blockquote></div><br></div>