[rabbitmq-discuss] Stale connections when using rabbitmq

Wed May 9 07:35:25 BST 2012

Hi,

I have a web server which communicates with my backend using rabbitmq.
The web server receives req at the rate of 3000/sec. I had a rabbitmq
server on 8GB i7 machine. I saw rabbitmq refusing further connections
after a while. When I did rabbitmqctl list_connections, I saw lot of
connections in 'blocked'/'blocking' state. I recently moved from
memcache based queue for want of async nature. Please suggest me ways
to debug this situation.

[Note: I reduced the RAM to 2GB to reproduce it quicker]

Listing connections ...
guest			blocked
guest			blocked
guest	172.16.223.114	51914	blocked
guest	172.16.223.113	33600	blocked
guest	172.16.223.114	46209	blocked
guest	172.16.223.114	53108	blocked
guest	172.16.223.114	52984	blocked
guest	172.16.223.114	52310	blocked
guest	172.16.223.114	51776	blocked
guest			blocked
guest	172.16.223.114	52390	blocked
guest			blocked
guest	172.16.223.114	53374	blocked
guest	172.16.223.114	53252	blocked
guest			blocked
guest	172.16.223.114	39293	blocking
guest	172.16.223.114	52772	blocked
guest	172.16.223.114	52000	blocked
guest			blocked
guest	172.16.223.114	53072	blocked
guest	172.16.223.114	52852	blocked
guest	172.16.223.114	51862	blocked
guest	172.16.223.114	53601	blocked
guest			blocked
guest	172.16.223.114	52258	blocked
guest			blocked
guest			blocked
guest	172.16.223.114	39286	blocked
guest	172.16.223.114	53200	blocked
guest			blocked
guest	172.16.223.114	51746	blocked
guest	172.16.223.114	52940	blocked
guest	172.16.223.114	32803	blocked
guest	172.16.223.114	53020	blocked
guest	172.16.223.114	52346	blocked

root at testmachine3:~/$ sudo rabbitmqctl list_connections | wc -l
830

sudo rabbitmqctl status
Status of node rabbit at testmachine3 ...
[{pid,939},
 {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.17"},
                        {os_mon,"CPO  CXC 138 46","2.2.5"},
                        {sasl,"SASL  CXC 138 11","2.1.9.3"},
                        {stdlib,"ERTS  CXC 138 10","1.17.3"},
                        {kernel,"ERTS  CXC 138 10","2.14.3"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B02 (erts-5.8.3) [source] [64-bit] [smp:
2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,1626268744},
          {processes,1296143128},
          {processes_used,1296128032},
          {system,330125616},
          {atom,3600969},
          {atom_used,3573207},
          {binary,6811184},
          {code,11216694},
          {ets,305365584}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1593409536}]
...done.

When my web server tries to connect to rabbitmq at this stage it give
me the following exception.

  File "/usr/local/lib/python2.7/dist-packages/msgbroker.py", line
194, in __init__
    self.connection =
pika.BlockingConnection(pika.ConnectionParameters(host=host_server))

  File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
blocking_connection.py", line 32, in __init__
    BaseConnection.__init__(self, parameters, None,
reconnection_strategy)

  File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
base_connection.py", line 50, in __init__
    reconnection_strategy)

  File "/usr/local/lib/python2.7/dist-packages/pika/connection.py",
line 170, in __init__
    self._connect()

  File "/usr/local/lib/python2.7/dist-packages/pika/connection.py",
line 228, in _connect
    self.parameters.port or  spec.PORT)

  File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
blocking_connection.py", line 44, in _adapter_connect
    self._handle_read()

  File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
base_connection.py", line 151, in _handle_read
    data = self.socket.recv(self._suggested_buffer_size)

timeout: timed out

Thanks,