[rabbitmq-discuss] How many queues can one broker support

[Simon MacMullen]

Hi Sharon.

There isn't a hard coded limit of course, but ~10M queues sounds 
optimistic, especially if combined with ~10M TCP connections.

I don't think 1) and 2) are all that different. Unless you need the 
wildcard matching of a topic exchange you might as well pick a direct 
exchange.

But in both cases you have millions of connections / consumers / queues. 
That *is* quite a lot to support on one machine. Are all these consumers 
going to connect simultaneously? You may have trouble raising the fd 
limit that high.

Also each connection / consumer / queue will take a few kb of memory 
even when idle - so that could eat up quite a few GB before you do anything.

My gut feeling is that supporting that many simultaneous connections 
will be a problem before the raw number of queues is.

Cheers, Simon

On 21/08/12 15:27, sharonbn wrote:
> Hello,
>
> I am investigating using RabbitMQ and AMQP to push notifications from our
> web app to client apps on Android devices.
>
> In our case, we have few publishers and many consumers (millions, perhaps
> ~10M), all connected to the broker and ready to consume messages.
> every now and then, the publisher needs to send one message to one and only
> one consumer.
> each consumer will receive few dozens of messages daily. Message
> acknowledgements is not mandatory.
> btw, the broker (or cluster?) will be hosted on AWS EC2 instances.
>
> I was wondering if there are any limitations on number of connected
> consumers and what's the best strategy to implement such a scenario.  From
> the documentation, it seems to me there are two ways to do this:
>
> first of all, each client app already reports a unique identifier to the web
> app (this is done prior to any message sending).  this unique id, known on
> both sides, can be used to ensure routing of the message to the specific
> client.
>
> 1) The most obvious way is to have one direct-exchange and create an
> exclusive queue per consumer with the unique id as the queue name.  this
> means there will be millions of queues that are most of the time empty or
> nearly empty.
>
> 2) Have one topic-exchange with (how many?) queues.  the unique id will be
> used as topic name.  Here we have millions of topic names.
>
> which is the best solutions (or perhaps a third option)?  any other
> considerations in this use case?  any special configuration/settings? any
> help is appreciated :)
>
> thanks,
>
> Sharon.
>
>
>
> --
> View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-many-queues-can-one-broker-support-tp21539.html
> Sent from the RabbitMQ mailing list archive at Nabble.com.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


-- 
Simon MacMullen
RabbitMQ, VMware