Mathias,<br><br>
<div class="gmail_quote">On Wed, May 19, 2010 at 1:20 AM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">Gustavo,
<div class="im"><br><br>Gustavo Aquino wrote:<br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">We put test to run for 1 hour, and monitoring queue size with rabbitmqctl list_queues, all the time list have 0 size, so consumer are getting all on the fly, and now we stop publisher and here is the BIG PROBLEM, consumer continue to consume messages from queue... and it's remain for 20 minutes after stop publisher......<br>
<br>If we stop consumer we can see the queue growing for long time..., therefore looking for it's we can conclude that our consumer are 20 minutes late behind publisher and it's a f... problem last message from publisher will be received by consumer just 20 minutes after.<br>
</blockquote><br></div>How are the messages being consumed? Using basic.consume? If so then the above observation is consistent with messages getting buffered in the network and/or consumers.<br><br></blockquote>
<div>Yes, basic consumer have other best way ? If I drop consumer queue continue to grow my friend, withour any consumers or publisher queue size continue to grow for 20 minutes after stop in this scenario.</div>
<div> </div>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">How are you monitoring the queue lengths? The default output of 'rabbitmqctl list_queues' will give a figure that is the sum of ready, unacknowledged and uncommitted messages. You can get a more detailed breakdown with s.t. like 'rabbitmqctl list_queues name messages_ready messages_unacknowledged messages_uncommitted' - see <a href="http://www.rabbitmq.com/admin-guide.html#list_queues" target="_blank">http://www.rabbitmq.com/admin-guide.html#list_queues</a>.<br>
</blockquote>
<div> </div>
<div>Thanks, I know this extra commands in rabbitmqctl list_queue, now we are using a lot of them.</div>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote"><br>If my above analysis is correct then for the total to be zero the clients would have to be consuming messages in no-ack mode, since otherwise the messages_unacknowledged count would go up. Are they?
<div class="im"><br></div></blockquote>
<div> </div>
<div>Rabbitmqctl just show message_ready size, other parameters like messages_unacknowledge and etc all the time are 0.</div>
<div> </div>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">
<div class="im"><br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">So can anyone help me ? Its's a common problem in Rabbit ? anybody here have this same problem ? We have talked about a possible exchange buffer cause and etc in this way, but to be sincerely in my thoughts ~3000 m/s is a bit easy to be processed for one message server.<br>
</blockquote><br></div>It looks like rabbit is handling the load just fine, but the consumers are struggling to keep up and are just buffering the messages. This can happen easily if the consumer code is employing s.t. like the QueueingConsumer in our Java client library - <a href="http://www.rabbitmq.com/releases/rabbitmq-java-client/v1.7.2/rabbitmq-java-client-javadoc-1.7.2/com/rabbitmq/client/QueueingConsumer.html" target="_blank">http://www.rabbitmq.com/releases/rabbitmq-java-client/v1.7.2/rabbitmq-java-client-javadoc-1.7.2/com/rabbitmq/client/QueueingConsumer.html</a><br>
<br>But it can also happen just through buffering in the network layers.<br></blockquote>
<div> </div>
<div>Itīs happened with consumers and without consumer... I donīt think that it as network problem.</div>
<div> </div>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote"><br>To avoid a large backlog building up, probably the easiest solution is to set a prefetch limit with basic.qos. But for that to be effective the consumers must be consuming in ack mode.<br>
<br></blockquote>
<div> </div>
<div>Thanks.</div>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote"><br>Regards,<br><font color="#888888"><br>Matthias.<br></font></blockquote></div><br>