[rabbitmq-discuss] RabbitMQ load issues

Tue Jan 17 12:47:26 GMT 2012

On 06/01/12 21:16, Jeremy Pierre wrote:
> Hi All,

Hi!

<snip>

> When we got to about 110,000 queues on the topic exchange, RabbitMQ
> basically stopped responding(on the direct exchange as well).  After a
> little while(10-15min), rabbitmqctl could no longer even connect to
> the still-running RabbitMQ instance(on the same machine).  RabbitMQ(2
> processes listed in top/htop that I could see) was using ~5-ish gigs
> of RAM.

Hmm. While RabbitMQ is designed to scale :-) 100k queues and 700k 
bindings does start to sound like "a lot".

Having said that I was able to create that number of queues / bindings 
on my desktop machine in half an hour or so without *too* much trouble.

Some tips:

* If you're declaring queues in bulk, make sure you do so synchonously 
(i.e. nowait=false).

* If you're going to have that many queues and that few routing keys 
then presumably every message published will be delivered to huge 
numbers of queues. I assume you expect your publishing rate to be small. 
Publishing a couple of messages per second is a lot if they're delivered 
to 50k queues.

* Every queue process consumes a fixed small amount of memory. Spreading 
queues out over a cluster will help.

* Every queue and binding creates a row in an Mnesia table. Mnesia 
tables always reside in RAM, so for huge numbers of queues in a cluster 
expect to use lots of RAM. Therefore instead of clustering you will 
probably find it better to shard the queues out over independent brokers 
and connect them with federation. The case of huge numbers of queues and 
very few routing keys could almost be made for federation. For the 
massive fanout case you probably want to build a (maybe multi-layered) 
federation tree rather than the complete graph people use more often 
with federation.

* For every active queue (one that has sent or received a message in the 
last few seconds) there is a small CPU cost when using the management 
plugin. With huge numbers of queues per node it may be helpful to 
disable mgmt. On the other hand "rabbitmqctl list_queues" will contact 
all the queues synchronously (maybe waking them up). Monitoring will be 
an issue.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware