[rabbitmq-discuss] Connection blocked by "flow" for more than 600 seconds

[Simon MacMullen]

OK, so your screenshot shows 750 queues and 753 connections. Was this 
from the same time as you had ~10k file descriptors in use? That sounds 
wrong.

I think your publishing connections are going into flow control because 
there's a squeeze on file descriptors, which is causing the queues to 
have to share a small number of file descriptors between them - thus 
slowing them down.

If you do have far more file descriptors in use than queues + 
connections, do you have any exotic plugins in use? What does "lsof -lnp 
<pid of server process>" say?

Cheers, Simon

On 11/10/2013 3:22AM, Choo wrote:
> Hi Simon,
>
> As memory is plenty, I found that file descriptors hit the default limit,
> so, I bumped the limit up to 5,120, and finally to 10,240 on each nodes.  It
> turned out that the file descriptors also touched the limit (around 10,086),
> and things started to go downhill.
>
> <http://rabbitmq.1065348.n5.nabble.com/file/n30402/ScreenShot.jpg>
>
> I started processes in reverse order, by starting subscriber-side first
> (1:42), then the bigger publishers later (1:45).  The number of published
> messages bounced up&down, then just after 1:48, the most of the publishers
> were blocked.
>
> There are more than 350 of blocked connections like below now (and file
> descriptors are running at 7,558 + 4,647 on 2 nodes):
> 10.95.212.11:33751 -> 10.95.212.13:5672	blocked	1261.558817	flow
> 10.95.212.11:33752 -> 10.95.212.13:5672	blocked	1326.324919	flow
> 10.95.212.11:33753 -> 10.95.212.13:5672	blocked	1326.45322	flow
> 10.95.212.11:33754 -> 10.95.212.13:5672	blocked	1278.581221	flow
> 10.95.212.11:33755 -> 10.95.212.13:5672	blocked	1312.584426	flow
> 10.95.212.11:33756 -> 10.95.212.13:5672	blocked	1279.623625	flow
> 10.95.212.11:33757 -> 10.95.212.13:5672	blocked	1294.492535	flow
> 10.95.212.11:33758 -> 10.95.212.13:5672	blocked	1276.134377	flow
> 10.95.212.11:33759 -> 10.95.212.13:5672	blocked	1292.862081	flow
> 10.95.212.11:33760 -> 10.95.212.13:5672	blocked	1290.695249	flow
> 10.95.212.11:33761 -> 10.95.212.13:5672	blocked	1255.599642	flow
> 10.95.212.11:33762 -> 10.95.212.13:5672	blocked	1284.984752	flow
>
> Please kindly suggest.
>
> Thank you and Best Regards,
> Choo
>
>
>
> --
> View this message in context: http://rabbitmq.1065348.n5.nabble.com/Connection-blocked-by-flow-for-more-than-600-seconds-tp30349p30402.html
> Sent from the RabbitMQ mailing list archive at Nabble.com.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

-- 
Simon MacMullen
RabbitMQ, Pivotal