[rabbitmq-discuss] Stale connections when using rabbitmq

[Simon MacMullen]

It sounds like you are overloading your RabbitMQ server - with 2.7.1 
messages would queue up indefinitely in process mailboxes, eventually 
filling up enough RAM that all connections will block. With 2.8.0+ 
individual connections will be throttled instead.

You might want to upgrade to 2.8.2 since that has some performance 
improvements anyway. You might also want to try turning hipe_compile on 
to get a performance boost. But if you are maxing out your server then 
ultimately you'll need bigger hardware or more machines.

Cheers, Simon

On 09/05/12 07:35, Venkat wrote:
> Hi,
>
> I have a web server which communicates with my backend using rabbitmq.
> The web server receives req at the rate of 3000/sec. I had a rabbitmq
> server on 8GB i7 machine. I saw rabbitmq refusing further connections
> after a while. When I did rabbitmqctl list_connections, I saw lot of
> connections in 'blocked'/'blocking' state. I recently moved from
> memcache based queue for want of async nature. Please suggest me ways
> to debug this situation.
>
> [Note: I reduced the RAM to 2GB to reproduce it quicker]
>
> Listing connections ...
> guest			blocked
> guest			blocked
> guest	172.16.223.114	51914	blocked
> guest	172.16.223.113	33600	blocked
> guest	172.16.223.114	46209	blocked
> guest	172.16.223.114	53108	blocked
> guest	172.16.223.114	52984	blocked
> guest	172.16.223.114	52310	blocked
> guest	172.16.223.114	51776	blocked
> guest			blocked
> guest	172.16.223.114	52390	blocked
> guest			blocked
> guest	172.16.223.114	53374	blocked
> guest	172.16.223.114	53252	blocked
> guest			blocked
> guest	172.16.223.114	39293	blocking
> guest	172.16.223.114	52772	blocked
> guest	172.16.223.114	52000	blocked
> guest			blocked
> guest	172.16.223.114	53072	blocked
> guest	172.16.223.114	52852	blocked
> guest	172.16.223.114	51862	blocked
> guest	172.16.223.114	53601	blocked
> guest			blocked
> guest	172.16.223.114	52258	blocked
> guest			blocked
> guest			blocked
> guest	172.16.223.114	39286	blocked
> guest	172.16.223.114	53200	blocked
> guest			blocked
> guest	172.16.223.114	51746	blocked
> guest	172.16.223.114	52940	blocked
> guest	172.16.223.114	32803	blocked
> guest	172.16.223.114	53020	blocked
> guest	172.16.223.114	52346	blocked
>
> root at testmachine3:~/$ sudo rabbitmqctl list_connections | wc -l
> 830
>
> sudo rabbitmqctl status
> Status of node rabbit at testmachine3 ...
> [{pid,939},
>   {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
>                          {mnesia,"MNESIA  CXC 138 12","4.4.17"},
>                          {os_mon,"CPO  CXC 138 46","2.2.5"},
>                          {sasl,"SASL  CXC 138 11","2.1.9.3"},
>                          {stdlib,"ERTS  CXC 138 10","1.17.3"},
>                          {kernel,"ERTS  CXC 138 10","2.14.3"}]},
>   {os,{unix,linux}},
>   {erlang_version,"Erlang R14B02 (erts-5.8.3) [source] [64-bit] [smp:
> 2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},
>   {memory,[{total,1626268744},
>            {processes,1296143128},
>            {processes_used,1296128032},
>            {system,330125616},
>            {atom,3600969},
>            {atom_used,3573207},
>            {binary,6811184},
>            {code,11216694},
>            {ets,305365584}]},
>   {vm_memory_high_watermark,0.4},
>   {vm_memory_limit,1593409536}]
> ...done.
>
> When my web server tries to connect to rabbitmq at this stage it give
> me the following exception.
>
>    File "/usr/local/lib/python2.7/dist-packages/msgbroker.py", line
> 194, in __init__
>      self.connection =
> pika.BlockingConnection(pika.ConnectionParameters(host=host_server))
>
>    File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
> blocking_connection.py", line 32, in __init__
>      BaseConnection.__init__(self, parameters, None,
> reconnection_strategy)
>
>    File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
> base_connection.py", line 50, in __init__
>      reconnection_strategy)
>
>    File "/usr/local/lib/python2.7/dist-packages/pika/connection.py",
> line 170, in __init__
>      self._connect()
>
>    File "/usr/local/lib/python2.7/dist-packages/pika/connection.py",
> line 228, in _connect
>      self.parameters.port or  spec.PORT)
>
>    File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
> blocking_connection.py", line 44, in _adapter_connect
>      self._handle_read()
>
>    File "/usr/local/lib/python2.7/dist-packages/pika/adapters/
> base_connection.py", line 151, in _handle_read
>      data = self.socket.recv(self._suggested_buffer_size)
>
> timeout: timed out
>
> Thanks,
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


-- 
Simon MacMullen
RabbitMQ, VMware