[rabbitmq-discuss] Consumer get blocked after all connections exhausted.

[Timon Wong]

Thanks Simon, it only appears during our boundary tests, we can
survival that by increasing the open files limit for now.


Best,
Timon Wong.


On Fri, May 16, 2014 at 7:46 PM, Simon MacMullen <simon at rabbitmq.com> wrote:
> Congratulations, you've found a bug.
>
> The way it's meant to work is that we ensure
> file-descriptors-used-for-sockets never exceeds a certain ratio of the total
> number of file descriptors, to make sure that we always have some left over
> for file operations.
>
> But there are some file operations (particularly around creating and
> deleting queues) which are counted as if they were sockets. So they block
> immediately when you hit the socket limit. They will tend to unblock as soon
> as you bounce off the socket limit, but if you stay stuck hard *at* the
> limit then they will not unblock.
>
> This has been wrong since 2.7.0, and will be fixed in 3.3.2 I guess.
>
> Cheers, Simon
>
>
> On 16/05/14 02:42, Timon Wang wrote:
>>
>> It seems that Consumer will get blocked after all connections exhausted.
>> Some rabbitmqctl commands such as `rabbitmqctl list_queues` and
>> `rabbitmqctl list_consumers` will get stucked as well.
>>
>> Here is an example to demonstrate that:
>> https://gist.github.com/timonwong/238f2379d1acbbd4ef0a
>> After all connections are exhausted, consumer will get blocked after
>> several seconds, `rabbitmqadmin list queues` will indicate that there
>> was unacknowledged messages in the queue:
>>
>>
>> +-------+-----------+-------------+----------------------+-----------+---------+------------------------+----------+----------+----------------+-------------------------+------------------+--------+---------+
>> | vhost |   name    | auto_delete | consumer_utilisation | consumers |
>> durable | exclusive_consumer_tag |  memory  | messages | messages_ready
>> | messages_unacknowledged |       node       | policy |  state  |
>>
>> +-------+-----------+-------------+----------------------+-----------+---------+------------------------+----------+----------+----------------+-------------------------+------------------+--------+---------+
>> | /     | test.msgs | False       | 0.105861902008       | 1         |
>> True    |                        | 74966600 | 46906    | 45664
>>   | 1242                    | rabbit at localhost |        | running |
>>
>> +-------+-----------+-------------+----------------------+-----------+---------+------------------------+----------+----------+----------------+-------------------------+------------------+--------+---------+
>>
>>
>> I've tried Python kombu library with both pyamqp and librabbitmq as
>> transport driver.
>>
>> Am I doing something wrong? Or is it a expected behavior?
>>
>> My environment
>>
>> Server side:
>> CentOS 6.3 64bit
>> RabbitMQ: 3.3.1
>>
>> Client side:
>> CentOS 6.3 64bit
>> Python 2.7.6
>> Kombu 3.0.16
>>
>> Thanks in advance.
>>
>> Regards,
>> Timon
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal