[rabbitmq-discuss] Large number of connections

[Valentino Volonghi]

On Apr 10, 2009, at 3:15 PM, Lawrence Kesteloot wrote:

> Hi Tony,
>
> Thanks for the talk on Wednesday.
>
> I'd rather not hack the kernel.

That's not hacking the kernel... If you want to setup a server to  
seriously handle
many thousands of connections you _have to_ set it up so that it is as  
fast as possible
to handle that traffic. The default parameters in the kernel are for  
desktop
and minimal RAM usage, on servers this doesn't make sense at all  
because you
already know how much traffic you are going to deal with and what the  
server is
going to be working on (DB, reverse proxy, load balancer, app server  
etc).

Notice also that every connection uses a port and you only have room  
for 65k
ports per interface, so for example one other setting that you need to  
add is
to remove the 2 minutes windows of TCP_WAIT. If you don't change these
parameters, mind you, you are going to have serious issues in  
production, for example
not enough ports to bind new sockets.

Here's a set of parameters that you might want to change in your kernel
to make it faster and better for many TCP connections.

# some spoof protection
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

# General gigabit tuning:
net.core.rmem_max = 8738000
net.core.wmem_max = 6553600
net.ipv4.tcp_rmem = 8192 873800 8738000
net.ipv4.tcp_wmem = 4096 655360 6553600

# VERY important to reuse ports in TCP_WAIT
net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 2500
vm.min_free_kbytes = 65536
vm.swappiness = 0

# Ports dedicated to clients from this server
net.ipv4.ip_local_port_range = 30000 65535

Furthermore you also might want to use ulimit to increase the number  
of allowed
fds per process and other parameters as tony already suggested.

--
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090410/1285cc8b/attachment.pgp