[rabbitmq-discuss] Consumer slows down (a lot) after producer exceeds a certain publish rate, why?

[Zhibo Wei]

Here is a very interesting chart.

Before 22:10, both publisher and consumer were both running in a stable
state (13000msgs/sec);
22:11, I increased the publish rate to 17000msgs/sec;
deliver rate went down while publish rate went up;
22:12, I shut down the publisher, the deliver rate went up high and drained
the all the messages left in queue in seconds.

I know it must be bound by something but I don't know what's that yet. CPU
and memory usage are just fine, and I put queue and consumer in the same
machine so probably bandwidth is not the root cause. Local I/O utilization
was low as well. What else could be the root cause? Any hint will be
appreciated.

btw, Everything looked good when I tested on my local machine
(20000+msgs/sec on both sides without any problem), but when I put them
onto EC2, this thing happens.

Thanks,
Zhibo


On Fri, Sep 27, 2013 at 3:05 PM, Zhibo Wei <zweicmu at gmail.com> wrote:

> Hi Alvaro,
>
> Thanks for pointing out this.
>
> But when I checked the connection status from management plugin, only
> publisher connections were in 'Flow/Blocked' states, all consumers'
> connections were still 'running' but in a very low rate.
>
> And, as far as I know, 15000 msgs/sec is still far from the ceiling (Based
> on
> http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/
> ).
>
> Did I miss anything?
>
> Thanks,
> Zhibo
>
>
> On Fri, Sep 27, 2013 at 2:52 PM, Alvaro Videla <videlalvaro at gmail.com>wrote:
>
>> Hi,
>>
>> Take a look at the flow control mechanism:
>> http://www.rabbitmq.com/memory.html
>>
>> "A per-connection mechanism that prevents messages being published faster
>> than they can be routed to queues." Perhaps your publishers are hitting
>> that.
>> Regards,
>>
>> Alvaro
>>
>>
>> On Fri, Sep 27, 2013 at 2:44 AM, Zhibo Wei <zweicmu at gmail.com> wrote:
>>
>>> 3.15 rabbitmq-server
>>> 3.15 java-client (consumer)
>>>  3.04 erlang (producer)
>>> ec2 m1.large
>>> 1 durable queue. no exchange.
>>> Producer has 5 connections, each connection holds 20 channels.
>>> Consumer has only 1 connection, who holds 20 channels, autoack = true.
>>>
>>> If the publish rate is equal or below 15000msgs/sec, then the consumer
>>> can hold it up (Queue never grows). However, if the publish rate exceeds
>>> 15,000, say 18,000 msgs/sec, then the delivery rate will drop to
>>> 400~2000/sec, then queue starts paging, blocks producer, and then
>>> everything crashes.
>>>
>>> The CPU and memory usages are just fine, but I'm not sure what else
>>> could cause such problem. Bandwidth? Socket buffer size?
>>>
>>> Anyone saw this kind of issue before? Any clues? Any other things I
>>> should check?
>>>
>>> Thanks,
>>> Zhibo
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130927/0deb8be1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbitmq.png
Type: image/png
Size: 42462 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130927/0deb8be1/attachment.png>