[rabbitmq-discuss] Consumer get blocked after all connections exhausted.

Mon May 19 02:08:50 BST 2014

Thanks Simon, it only appears during our boundary tests, we can
survival that by increasing the open files limit for now.

Best,
Timon Wong.

On Fri, May 16, 2014 at 7:46 PM, Simon MacMullen <simon at rabbitmq.com> wrote:
> Congratulations, you've found a bug.
>
> The way it's meant to work is that we ensure
> file-descriptors-used-for-sockets never exceeds a certain ratio of the total
> number of file descriptors, to make sure that we always have some left over
> for file operations.
>
> But there are some file operations (particularly around creating and
> deleting queues) which are counted as if they were sockets. So they block
> immediately when you hit the socket limit. They will tend to unblock as soon
> as you bounce off the socket limit, but if you stay stuck hard *at* the
> limit then they will not unblock.
>
> This has been wrong since 2.7.0, and will be fixed in 3.3.2 I guess.
>
> Cheers, Simon
>
>
> On 16/05/14 02:42, Timon Wang wrote:
>>
>> It seems that Consumer will get blocked after all connections exhausted.
>> Some rabbitmqctl commands such as `rabbitmqctl list_queues` and
>> `rabbitmqctl list_consumers` will get stucked as well.
>>
>> Here is an example to demonstrate that:
>> https://gist.github.com/timonwong/238f2379d1acbbd4ef0a
>> After all connections are exhausted, consumer will get blocked after
>> several seconds, `rabbitmqadmin list queues` will indicate that there
>> was unacknowledged messages in the queue:
>>
>>
>> +-------+-----------+-------------+----------------------+-----------+---------+------------------------+----------+----------+----------------+-------------------------+------------------+--------+---------+
>> | vhost |   name    | auto_delete | consumer_utilisation | consumers |
>> durable | exclusive_consumer_tag |  memory  | messages | messages_ready
>> | messages_unacknowledged |       node       | policy |  state  |
>>
>> +-------+-----------+-------------+----------------------+-----------+---------+------------------------+----------+----------+----------------+-------------------------+------------------+--------+---------+
>> | /     | test.msgs | False       | 0.105861902008       | 1         |
>> True    |                        | 74966600 | 46906    | 45664
>>   | 1242                    | rabbit at localhost |        | running |
>>
>> +-------+-----------+-------------+----------------------+-----------+---------+------------------------+----------+----------+----------------+-------------------------+------------------+--------+---------+
>>
>>
>> I've tried Python kombu library with both pyamqp and librabbitmq as
>> transport driver.
>>
>> Am I doing something wrong? Or is it a expected behavior?
>>
>> My environment
>>
>> Server side:
>> CentOS 6.3 64bit
>> RabbitMQ: 3.3.1
>>
>> Client side:
>> CentOS 6.3 64bit
>> Python 2.7.6
>> Kombu 3.0.16
>>
>> Thanks in advance.
>>
>> Regards,
>> Timon
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal