<div dir="ltr"><div><div><div>Here is a very interesting chart.<br><br></div>Before 22:10, both publisher and consumer were both running in a stable state (13000msgs/sec);<br></div>22:11, I increased the publish rate to 17000msgs/sec;<br>
</div><div>deliver rate went down while publish rate went up;<br></div><div>22:12, I shut down the publisher, the deliver rate went up high and drained the all the messages left in queue in seconds.<br><br></div><div>I know it must be bound by something but I don't know what's that yet. CPU and memory usage are just fine, and I put queue and consumer in the same machine so probably bandwidth is not the root cause. Local I/O utilization was low as well. What else could be the root cause? Any hint will be appreciated.<br>
<br></div><div>btw, Everything looked good when I tested on my local machine (20000+msgs/sec on both sides without any problem), but when I put them onto EC2, this thing happens.<br><br></div><div>Thanks,<br>Zhibo<br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Sep 27, 2013 at 3:05 PM, Zhibo Wei <span dir="ltr"><<a href="mailto:zweicmu@gmail.com" target="_blank">zweicmu@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hi Alvaro,<br><br>Thanks for pointing out this.<br><br></div>But when I checked the connection status from management plugin, only publisher connections were in 'Flow/Blocked' states, all consumers' connections were still 'running' but in a very low rate.<br>
<br></div><div>And, as far as I know, 15000 msgs/sec is still far from the ceiling (Based on <a href="http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/" target="_blank">http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/</a>).<br>
<br></div><div>Did I miss anything?<br></div><div><br></div>Thanks,<br>Zhibo<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Sep 27, 2013 at 2:52 PM, Alvaro Videla <span dir="ltr"><<a href="mailto:videlalvaro@gmail.com" target="_blank">videlalvaro@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Take a look at the flow control mechanism: <a href="http://www.rabbitmq.com/memory.html" target="_blank">http://www.rabbitmq.com/memory.html</a></div>
<div>
<p>"A per-connection mechanism that prevents messages being published faster than they can be routed to queues." Perhaps your publishers are hitting that. </p></div><div>Regards,</div><div><br></div><div>
Alvaro</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Fri, Sep 27, 2013 at 2:44 AM, Zhibo Wei <span dir="ltr"><<a href="mailto:zweicmu@gmail.com" target="_blank">zweicmu@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div><div><div><div><div>3.15 rabbitmq-server<br></div><div>3.15 java-client (consumer)<br>
</div><div>
3.04 erlang (producer)<br></div><div>ec2 m1.large<br>1 durable queue. no exchange.<br></div>Producer has 5 connections, each connection holds 20 channels.<br>
</div>Consumer has only 1 connection, who holds 20 channels, autoack = true.<br><br></div>If the publish rate is equal or below 15000msgs/sec, then the consumer can hold it up (Queue never grows). However, if the publish rate exceeds 15,000, say 18,000 msgs/sec, then the delivery rate will drop to 400~2000/sec, then queue starts paging, blocks producer, and then everything crashes.<br>
<br></div>The CPU and memory usages are just fine, but I'm not sure what else could cause such problem. Bandwidth? Socket buffer size?<br><br></div>Anyone saw this kind of issue before? Any clues? Any other things I should check?<br>
<br></div>Thanks,<br>Zhibo<br></div>
<br></div></div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>