[rabbitmq-discuss] RabbitMQ load issues

Fri Jan 20 12:32:58 GMT 2012

On 17/01/12 20:57, Jeremy Pierre wrote:
>> * If you're declaring queues in bulk, make sure you do so synchonously (i.e.
>> nowait=false).
>
> To be perfectly honest, I can't find this (with a relatively cursory
> investigation) in the APIs I'm using(Akka 1.2 AMQP module that
> leverages RabbitMQ's Java lib iirc).  Having said that, I'm creating
> queues one at a time(albeit quickly) - not sure if this addresses the
> bulk queue declaration you're referring to?  I'll dig further into the
> APIs on my side.

Synchronous queue.declare (i.e. with a response) is the default in most 
APIs - in general you want to know when the queue is declared.

>> * If you're going to have that many queues and that few routing keys then
>> presumably every message published will be delivered to huge numbers of
>> queues. I assume you expect your publishing rate to be small. Publishing a
>> couple of messages per second is a lot if they're delivered to 50k queues.
>
> The number of routing keys is actually fairly high given that the
> userId and actual message categories are completely variable.  With
> the example I gave(~100k queues), each queue is bound to a unique
> routing key(topic) and a message typically is delivered to a very
> small number of queues(20-30 would be a theoretical maximum and a very
> rare situation).  As additional applications are built on top of the
> base collection layer and RabbitMQ, the queue count will go much
> higher(obviously).
>
> For what it's worth, our testing load(30k "users", 6 queues per user
> in my original posting though we never got that high before rabbit
> choked) generates an easy 150-200 messages per second going into
> RabbitMQ and this can spike higher.  Production will see 200k-300k
> users at least I expect.

Ah OK. 200 msg/s in is fine as long as they're not going to 50k queues 
each :)

>> * Every queue process consumes a fixed small amount of memory. Spreading
>> queues out over a cluster will help.
>
> We're experimenting with a 3-node cluster as of today though I've
> reduced the queue count by determining the update type at my
> application layer and using topics with wildcards, e.g.
> "<source>.<userId>.*"
>
>> * Every queue and binding creates a row in an Mnesia table. Mnesia tables
>> always reside in RAM, so for huge numbers of queues in a cluster expect to
>> use lots of RAM. Therefore instead of clustering you will probably find it
>> better to shard the queues out over independent brokers and connect them
>> with federation. The case of huge numbers of queues and very few routing
>> keys could almost be made for federation. For the massive fanout case you
>> probably want to build a (maybe multi-layered) federation tree rather than
>> the complete graph people use more often with federation.
>
> Thanks for suggesting this!  We may go this route but I'm wary of
> putting sharding logic in our application layer(perhaps unjustifiably
> so).  Truth be told I was kind of hoping for a "silver bullet" with
> RabbitMQ ;)  In the meantime, we'll bump the RAM for our cluster nodes
> and see what happens.

Hopefully you should not need (much) sharding logic in your app - just 
enough to pick a broker to connect to (which you'd need for clustering 
anyway). Set up the federation correctly and you could just publish to 
one broker / an arbitrary broker and get the messages routed accordingly.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware