[rabbitmq-discuss] Running into some issues with high queue counts(again)

Thu Apr 5 19:02:49 BST 2012

Hi all,

I seem to be hitting some limits with RabbitMQ queue volumes again, here's 
the rough breakdown of the situation for the purposes of my current testing:

One RabbitMQ node, 8 cores, 32GB RAM - Rabbit version 2.8.1-1, Erlang R13B 
on Ubuntu 10.04
2 nodes connecting and trying to create 150k queues each on a topic exchange
10 routing keys

The 2 nodes that connect run code written for Akka 2.0 with the development 
AMQP module, Scala 2.9.1, OpenJDK-6

Creating ~200k queues goes fine, only takes ~5min.  Past this, I get a lot 
of timing out on channel/queue creation(timeout value bumped up to 60 
seconds).

The use case is essentially some realtime-ish event based stuff for web 
clients(will be done using Play! 2.0).  Under the current plan, any given 
user will create at least 2 queues:  one for general events that occur 
during their session and one for each category of data they're 
receiving(e.g. a chat session, updates from various data sources/streams). 
 The 300k queues I'm attempting to create is a much smaller count than we'd 
eventually be running in production.

So the question is, am I approaching this the wrong way?  Should I be 
treating RabbitMQ as a more general message bus and performing more complex 
routing on the non-RabbitMQ web server/application nodes?  Any and all 
feedback would be most welcome.

Regards,

Jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120405/697cf55d/attachment.htm>