[rabbitmq-discuss] Millions of Queues

[Ryan Williams (Which)]

Hey all, we have a different use case for rabbit than usual.  We'd like 
to create a rabbit cluster that handles at least 250,000 queues, with at 
leat 1 million total bindings; ideally we would have 20+ million and 80+ 
million respectively.  Basically we want infinite scalability along the 
# of queues axis.  We've been testing this configuration by using a 
modified ScalabilityTest that simply created oodles of queues with 4 
bindings each.  We observed that we were able to get up around 200,000 
queues before the test would halt.  At this time the server was 
consuming a large amount of physical memory and was entering swap.

We repeated the experiment with a cluster of two machines and achieved 
basically the same result -- except the "primary" machine (the one 
ScalabilityTest was interacting with) had most of its memory consumed, 
and the "secondary" machine had 40% of its memory consumed by beam 
processes.  Would we have achieved better results by manually targeting 
ScalabilityTest at the secondary machine as well?

Are we doing something wrong in our setup here?  What's the maximum 
number of queues that has been achieved by anyone on this list, and how 
did you get there?

I see that in this email: 
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2008-October/002150.html 
, Ben Hood mentions that routing complexity is O(n), where n is the 
number of bindings, which doesn't bode well for our particular 
application.  Assuming that I'm interpreting that correctly, is there 
anything we can do to tackle that problem to enable huge numbers of queues?