[rabbitmq-discuss] How many queues can one broker support

[Brett Cameron]

Sharon,

Further to Simon's comments, each connection will consume ~66K memory and
each connection also creates 8 Erlang processes (or thereabouts), so you'd
need to factor in this memory usage and ensure that Erlang is able to spin
up the required number of processes. You'd obviously also need to tweak a
few system parameters to allow that many connections/sockets - all quite
doable of course, but I'd concur with Simon that unless you have a pretty
grunty server you could face some challenges. That being said, Erlang (and
by inference RabbitMQ) can certainly handle this sort of thing... you might
like to have a look at http://www.youtube.com/watch?v=wDk6l3tPBuw.

Brett


On Wed, Aug 22, 2012 at 4:16 AM, Simon MacMullen <simon at rabbitmq.com> wrote:

> Hi Sharon.
>
> There isn't a hard coded limit of course, but ~10M queues sounds
> optimistic, especially if combined with ~10M TCP connections.
>
> I don't think 1) and 2) are all that different. Unless you need the
> wildcard matching of a topic exchange you might as well pick a direct
> exchange.
>
> But in both cases you have millions of connections / consumers / queues.
> That *is* quite a lot to support on one machine. Are all these consumers
> going to connect simultaneously? You may have trouble raising the fd limit
> that high.
>
> Also each connection / consumer / queue will take a few kb of memory even
> when idle - so that could eat up quite a few GB before you do anything.
>
> My gut feeling is that supporting that many simultaneous connections will
> be a problem before the raw number of queues is.
>
> Cheers, Simon
>
>
> On 21/08/12 15:27, sharonbn wrote:
>
>> Hello,
>>
>> I am investigating using RabbitMQ and AMQP to push notifications from our
>> web app to client apps on Android devices.
>>
>> In our case, we have few publishers and many consumers (millions, perhaps
>> ~10M), all connected to the broker and ready to consume messages.
>> every now and then, the publisher needs to send one message to one and
>> only
>> one consumer.
>> each consumer will receive few dozens of messages daily. Message
>> acknowledgements is not mandatory.
>> btw, the broker (or cluster?) will be hosted on AWS EC2 instances.
>>
>> I was wondering if there are any limitations on number of connected
>> consumers and what's the best strategy to implement such a scenario.  From
>> the documentation, it seems to me there are two ways to do this:
>>
>> first of all, each client app already reports a unique identifier to the
>> web
>> app (this is done prior to any message sending).  this unique id, known on
>> both sides, can be used to ensure routing of the message to the specific
>> client.
>>
>> 1) The most obvious way is to have one direct-exchange and create an
>> exclusive queue per consumer with the unique id as the queue name.  this
>> means there will be millions of queues that are most of the time empty or
>> nearly empty.
>>
>> 2) Have one topic-exchange with (how many?) queues.  the unique id will be
>> used as topic name.  Here we have millions of topic names.
>>
>> which is the best solutions (or perhaps a third option)?  any other
>> considerations in this use case?  any special configuration/settings? any
>> help is appreciated :)
>>
>> thanks,
>>
>> Sharon.
>>
>>
>>
>> --
>> View this message in context: http://rabbitmq.1065348.n5.**
>> nabble.com/How-many-queues-**can-one-broker-support-**tp21539.html<http://rabbitmq.1065348.n5.nabble.com/How-many-queues-can-one-broker-support-tp21539.html>
>> Sent from the RabbitMQ mailing list archive at Nabble.com.
>> ______________________________**_________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
>> https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>>
>
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>
> ______________________________**_________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
> https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120822/cce46501/attachment.htm>