[rabbitmq-discuss] Large number of connections

[Lawrence Kesteloot]

Hi Tony,

Thanks for the talk on Wednesday.

I'd rather not hack the kernel. If RabbitMQ can already handle the queues
and messages, could I instead have a dozen multiplexer processes that do
nothing but multiplex 1000 TCP AMQP connections into a single one? I don't
know if the AMQP protocol makes this possible. It might need to keep track
of virtual connections within the single connection.

Lawrence



On Fri, Apr 10, 2009 at 2:53 PM, Tony Garnock-Jones <tonyg at lshift.net>wrote:

> Hi Lawrence,
>
> Lawrence Kesteloot wrote:
>
>> I need to have a large number of connections to the message broker,
>> perhaps 10,000. There is typically (on Unix) a limit of 1024 open file
>> descriptors per process.
>>
>
> Indeed. You will need to engage in kernel (or at least rlimit) tuning to
> raise the limit. (You may also need to increase the limit the erlang VM
> places on the maximum number of processes; "+P 1000000" argument to the VM)
>
> I've had upwards of 4000 simultaneous connections to a single node before,
> but haven't done any serious testing of it with so many connections.
>
>  Multiple instances of the broker?
>>
>
> Clustering will help spread the load, so yes, maybe.
>
>  Also, each connection will
>> have its own queue. Can RabbitMQ deal with tens of thousands of
>> queues? The total message throughput will be on the order of the
>> number of connections per second (several thousand per second).
>>
>
> It can deal with tens or even hundreds of thousands of queues. The
> throughput may be a limiting factor, but again clustering may help there.
>
> Regards,
>  Tony
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090410/3e9ec6ae/attachment.htm