[rabbitmq-discuss] Per Connection Flow Control Issue

[Brendan Hay]

Just to follow up with this, the issue was due to
GRO<http://lwn.net/Articles/358910/> being
turned on the NIC for our load balancers, which are running an older
version of LVS (prior to
2.6.39<http://archive.linuxvirtualserver.org/html/lvs-users/2011-11/msg00024.html>
).

In essence, this caused the high throughput publishers to have their data
streams fragmented and they fell further behind with the observed send
queue overflow on the clients. We'd previously seen something similar with
large HTTP streaming uploads.

GRO can be turned off using: ethtool -K <NIC> gro off

Cheers,
Brendan

(It's not the flow control, I know, I know ..)




On Fri, Oct 26, 2012 at 2:41 PM, Simon MacMullen <simon at rabbitmq.com> wrote:

> On 26/10/12 11:23, Brendan Hay wrote:
>
>> Yes, so what should the expected observation be - the client code
>> carries on publishing into a black hole,
>> which means the send queue on the client/peer socket should keep
>> growing,
>>
>
> Well, the send queue is limited in size. So the publisher should block
> fairly quickly.
>
>
>  and when the rabbit reader is
>> issued new credits, it will lap it all up?
>>
>
> Yes.
>
>
>  It just seems the 'flow' status in the UI stays on permanently, it
>> doesn't seem to be toggling at high speed, just locked.
>>
>
> Well, the flow status *in the UI* is designed not to toggle at high speed,
> since that would not be very readable - it shows "flow" if the connection
> has been blocked in the last 5 seconds. This is driven by
> "last_blocked_age" and "last_blocked_by".
>
> The rabbitmqctl command results you posted show some connections which had
> blocked some time in the past, all more than 5 seconds ago. Unfortunately I
> forgot to ask you to add "state" to the list of columns, to determine if
> they were blocked now. If we see any connections that were blocked by flow
> control a long time ago, and are still blocked, then I'm concerned.
>
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121120/3f728d7f/attachment.htm>