<div dir="ltr"><div><div>Ah....Maybe that's why<br><br></div>I will do more investigations on the CPU usage for each core and split all msgs into several different queues to see if that will help.<br><br></div>Thanks a lot Simon, It's really helpful.<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Sep 30, 2013 at 2:16 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">While you say that CPU usage is fine, bear in mind that a single queue can only use one CPU core at once.<br>
<br>
It is quite possible that you are simply giving the queue too much work to do. (And you're not hitting its limit when testing on your own machine because that's faster than a large EC2 instance.)<br>
<br>
Unfortunately when a queue is CPU-limited we don't guarantee that it will prioritise delivering messages to consumers over accepting messages from publishers. We used to, but this can have undesirable consequences in other contexts (a large queue which suddenly got consumers could use 100% of its CPU draining, thus blocking publishers for potentially a long time). So in particular if you have a decent number of publishers it's possible to just overwhelm the queue.<br>
<br>
If you are able to split your work up across several queues I suspect that would help a lot.<br>
<br>
Cheers, Simon<div class="im"><br>
<br>
On 27/09/2013 23:43, Zhibo Wei wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Here is a very interesting chart.<br>
<br>
Before 22:10, both publisher and consumer were both running in a stable<br>
state (13000msgs/sec);<br>
22:11, I increased the publish rate to 17000msgs/sec;<br>
deliver rate went down while publish rate went up;<br>
22:12, I shut down the publisher, the deliver rate went up high and<br>
drained the all the messages left in queue in seconds.<br>
<br>
I know it must be bound by something but I don't know what's that yet.<br>
CPU and memory usage are just fine, and I put queue and consumer in the<br>
same machine so probably bandwidth is not the root cause. Local I/O<br>
utilization was low as well. What else could be the root cause? Any hint<br>
will be appreciated.<br>
<br>
btw, Everything looked good when I tested on my local machine<br>
(20000+msgs/sec on both sides without any problem), but when I put them<br>
onto EC2, this thing happens.<br>
<br>
Thanks,<br>
Zhibo<br>
<br>
<br>
On Fri, Sep 27, 2013 at 3:05 PM, Zhibo Wei <<a href="mailto:zweicmu@gmail.com" target="_blank">zweicmu@gmail.com</a><br></div><div class="im">
<mailto:<a href="mailto:zweicmu@gmail.com" target="_blank">zweicmu@gmail.com</a>>> wrote:<br>
<br>
Hi Alvaro,<br>
<br>
Thanks for pointing out this.<br>
<br>
But when I checked the connection status from management plugin,<br>
only publisher connections were in 'Flow/Blocked' states, all<br>
consumers' connections were still 'running' but in a very low rate.<br>
<br>
And, as far as I know, 15000 msgs/sec is still far from the ceiling<br>
(Based on<br>
<a href="http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/" target="_blank">http://www.rabbitmq.com/blog/<u></u>2012/04/25/rabbitmq-<u></u>performance-measurements-part-<u></u>2/</a>).<br>
<br>
Did I miss anything?<br>
<br>
Thanks,<br>
Zhibo<br>
<br>
<br>
On Fri, Sep 27, 2013 at 2:52 PM, Alvaro Videla<br></div><div class="im">
<<a href="mailto:videlalvaro@gmail.com" target="_blank">videlalvaro@gmail.com</a> <mailto:<a href="mailto:videlalvaro@gmail.com" target="_blank">videlalvaro@gmail.com</a>><u></u>> wrote:<br>
<br>
Hi,<br>
<br>
Take a look at the flow control mechanism:<br>
<a href="http://www.rabbitmq.com/memory.html" target="_blank">http://www.rabbitmq.com/<u></u>memory.html</a><br>
<br>
"A per-connection mechanism that prevents messages being<br>
published faster than they can be routed to queues." Perhaps<br>
your publishers are hitting that.<br>
<br>
Regards,<br>
<br>
Alvaro<br>
<br>
<br>
On Fri, Sep 27, 2013 at 2:44 AM, Zhibo Wei <<a href="mailto:zweicmu@gmail.com" target="_blank">zweicmu@gmail.com</a><br></div><div class="im">
<mailto:<a href="mailto:zweicmu@gmail.com" target="_blank">zweicmu@gmail.com</a>>> wrote:<br>
<br>
3.15 rabbitmq-server<br>
3.15 java-client (consumer)<br>
3.04 erlang (producer)<br>
ec2 m1.large<br>
1 durable queue. no exchange.<br>
Producer has 5 connections, each connection holds 20 channels.<br>
Consumer has only 1 connection, who holds 20 channels,<br>
autoack = true.<br>
<br>
If the publish rate is equal or below 15000msgs/sec, then<br>
the consumer can hold it up (Queue never grows). However, if<br>
the publish rate exceeds 15,000, say 18,000 msgs/sec, then<br>
the delivery rate will drop to 400~2000/sec, then queue<br>
starts paging, blocks producer, and then everything crashes.<br>
<br>
The CPU and memory usages are just fine, but I'm not sure<br>
what else could cause such problem. Bandwidth? Socket buffer<br>
size?<br>
<br>
Anyone saw this kind of issue before? Any clues? Any other<br>
things I should check?<br>
<br>
Thanks,<br>
Zhibo<br>
<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br></div>
<mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@<u></u>lists.rabbitmq.com</a>><div class="im"><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br></div>
<mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@<u></u>lists.rabbitmq.com</a>><div class="im"><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
<br>
</div></blockquote>
<br>
</blockquote></div><br></div>