[rabbitmq-discuss] RabbitMQ with many queues

[Emile Joubert]

Hi Eric,

On 18/04/13 14:23, Eric Berg wrote:
> Thanks for the reply Emile. Right now we have a 4 node cluster with
> primary and failover haproxy round robin load balancers, so dynamic
> generation of queues tend to be pretty well distributed especially with
> a 2 node ha policy. I guess it will just take some close monitoring to
> verify that the resources are not being exhausted, though even adding
> new nodes to the cluster would not help in that case correct? Since in a
> 4 node cluster with a 2 node ha policy, only newly created queues would
> be replicated to the new node? How do others handle situations like this?

Queues won't mirror themselves on a new node spontaneously, so the
opportunities for rebalancing is limited to queue creation and node
death at the moment. In the case of an "exactly" policy the list of
candidate nodes is randomised when mirror nodes are picked, so that
offers some opportunity for rebalancing by restarting nodes.

If you want to guarantee that a node is mirrored on a particular new
node then you can use the "nodes" policy and spell out exactly which
nodes the queue should live on.

> - PHP -
> On a related note, we have had some unfortunate experiences when testing
> a few different php libraries and Rabbit kicks in its flow control. It
> looks like that our producers using various php libs in an Apache
> setting dont handle the flow control well, while the php connection dies
> Rabbit still thinks they are active even though no producing is coming
> across, filling up the file descriptors. 

The flow-control as implemented by the broker should not depend on the
client library, as it works on the network level. It is possible that
you see old blocked connections in the output of "rabbitmqctl
list_connections". This problem is not limited to PHP, it is a general
problem that we have not yet found a solution to.



-Emile