[rabbitmq-discuss] Millions of Queues

[Matthias Radestock]

Ryan,

Ryan Williams (Which) wrote:
> Hey all, we have a different use case for rabbit than usual.  We'd like 
> to create a rabbit cluster that handles at least 250,000 queues, with at 
> leat 1 million total bindings;

That should be possible on a single machine with a decent amount of 
memory - just.

> ideally we would have 20+ million and 80+ million respectively.
> Basically we want infinite scalability along the # of queues axis.

One problem you are going to run into here is that while queue processes 
reside on single nodes (and hence adding more nodes gives you more room 
to host the queue processes), all the routing info - i.e. queue, 
exchange and binding records - is held in memory on each node. 80 
million binding records are unlikely to fit into physical memory.

> We repeated the experiment with a cluster of two machines and achieved 
> basically the same result -- except the "primary" machine (the one 
> ScalabilityTest was interacting with) had most of its memory consumed, 
> and the "secondary" machine had 40% of its memory consumed by beam 
> processes.  Would we have achieved better results by manually targeting 
> ScalabilityTest at the secondary machine as well?

What you are observing here follows directly from the explanation above 
- the queue and binding records will consume memory on both machines 
whereas the queue processes will only consume memory on the machine on 
which they were created. So by targeting ScalabilityTest at both 
machines you'd be able to balance the memory usage.

> Are we doing something wrong in our setup here?  What's the maximum 
> number of queues that has been achieved by anyone on this list, and how 
> did you get there?

I am pretty sure Ben managed to get to 100s of thousands of queues and 
bindings in his tests on a single node.

> I see that in this email: 
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2008-October/002150.html 
> , Ben Hood mentions that routing complexity is O(n), where n is the 
> number of bindings, which doesn't bode well for our particular 
> application.  Assuming that I'm interpreting that correctly, is there 
> anything we can do to tackle that problem to enable huge numbers of queues?

Ben was referring to topic exchanges. For direct exchanges the routing 
cost is linear (or possibly O(n * log n)) in the number of *matching* 
bindings. Does your application definitely require the use of topic 
exchanges or would direct exchanges be sufficient?

Also, regarding the 20/80 million queues/bindings, would it be possible 
to partition these s.t. rather than having a single RabbitMQ cluster 
with that number of queues/bindings you could have n individual RabbitMQ 
servers, each with 1/nth of the queues/bindings? That might involve your 
producers having to publish messages to more than one broker, and 
consumers consuming from more than one broker, but depending on the 
exact nature of your app that may not be too arduous.


Matthias.