[rabbitmq-discuss] Consumer stop to receive messages but continue listening queue problem.

Wed May 19 05:20:10 BST 2010

Gustavo,

Gustavo Aquino wrote:
> We put test to run for 1 hour, and monitoring queue size with 
> rabbitmqctl list_queues, all the time list have 0 size, so consumer are 
> getting all on the fly, and now we stop publisher and here is the BIG 
> PROBLEM, consumer continue to consume messages from queue... and it's 
> remain for 20 minutes after stop publisher......
> 
> If we stop consumer we can see the queue growing for long time..., 
> therefore looking for it's we can conclude that our consumer are 20 
> minutes late behind publisher and it's a f... problem last message from 
> publisher will be received by consumer just 20 minutes after.

How are the messages being consumed? Using basic.consume? If so then the 
above observation is consistent with messages getting buffered in the 
network and/or consumers.

How are you monitoring the queue lengths? The default output of 
'rabbitmqctl list_queues' will give a figure that is the sum of ready, 
unacknowledged and uncommitted messages. You can get a more detailed 
breakdown with s.t. like 'rabbitmqctl list_queues name messages_ready 
messages_unacknowledged messages_uncommitted' - see 
http://www.rabbitmq.com/admin-guide.html#list_queues.

If my above analysis is correct then for the total to be zero the 
clients would have to be consuming messages in no-ack mode, since 
otherwise the messages_unacknowledged count would go up. Are they?

> So can anyone help me ? Its's a common problem in Rabbit ? anybody here 
> have this same problem ? We have talked about a possible exchange buffer 
> cause and etc in this way, but to be sincerely in my thoughts ~3000 m/s 
> is a bit easy to be processed for one message server.

It looks like rabbit is handling the load just fine, but the consumers 
are struggling to keep up and are just buffering the messages. This can 
happen easily if the consumer code is employing s.t. like the 
QueueingConsumer in our Java client library - 
http://www.rabbitmq.com/releases/rabbitmq-java-client/v1.7.2/rabbitmq-java-client-javadoc-1.7.2/com/rabbitmq/client/QueueingConsumer.html

But it can also happen just through buffering in the network layers.

To avoid a large backlog building up, probably the easiest solution is 
to set a prefetch limit with basic.qos. But for that to be effective the 
consumers must be consuming in ack mode.

Regards,

Matthias.