[rabbitmq-discuss] RabbitMQ cluster with potentially millions of connections

[McIntosh Jason]

Romanas:

Umm, I'm reasonably sure millions of persistent connections isn't possible through a single entry point.  First, with TCP/IP communications an interface only has 65k ports (roughly).  Of those, quite a few are typically in use for things like SSH requests or similar.  I'm actually surprised you made it to 30k simultaneous connections.  Now, you should be able to handle that many channels when distributed across a cluster - at least I think so though I've not really tested that situation.  The kicker is even if a server can handle say 30k connections and so you have your 50 servers, the load balancer itself would become your bottle neck as it couldn't keep open a million persistent connections (again, 65k limit, which is really a lot lower).  Most load balancers act as a proxy, not as a DNS lookup in concept.

Second, even on a single connection, with only a few channels, with my testing on a newer machine with only 8GB of RAM, and 4 cores, I was able to push through 20+k messages a second easily.  With HIPE turned on I was pushing closer to 30k messages a second (these were very small messages though for testing purposes).  The the performance on this with durable messages seems to be very dependent on message size.

Jason

On Jan 26, 2013, at 5:58 AM, Romanas S. wrote:

> Hi guys,
>  I'm trying to come up with an architecture for a RabbitMQ cluster that could potentially handle millions of (persistent) connections. I've set up a cluster of Rabbits with 3 nodes (16GB RAM, 4x cpu cores each) and a Load Balancer in front of them, increased all of the system limits on sockets and file descriptors, but can't seem to scale to more then 30k connections on the entire cluster. 
> 
> Each connections has it's own exchange and a temporary direct queue, RAM usage is about 2-3GB on each node and CPU usage peaks at about 200% (which on 4 cores isn't all that bad). At first all is quite well and the cluster manages upwards of 1k msgs/second (I haven't tried more, pretty sure it would handle it). However, after scaling to that many connections, the cluster seems to grind to a halt and eventually nodes become unresponsive. The management API takes upwards of 5 (FIVE!) minutes to retrieve a server summary report. Am I missing something here? Or is this just a bad idea altogether? I was expecting that if each node was able to handle 30k+ connections, I could have a cluster of 50 or so (with multiple balancers, of course!) and live a happy life  ;-)
> 
> Thanks,
> Roman
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss