[rabbitmq-discuss] Per Connection Flow Control Issue

[Simon MacMullen]

Thanks for getting back to us, good to know it wasn't flow control.

Cheers, Simon

On 20/11/12 06:44, Brendan Hay wrote:
> Just to follow up with this, the issue was due to GRO
> <http://lwn.net/Articles/358910/> being turned on the NIC for our load
> balancers, which are running an older version of LVS (prior to 2.6.39
> <http://archive.linuxvirtualserver.org/html/lvs-users/2011-11/msg00024.html>).
>
> In essence, this caused the high throughput publishers to have their
> data streams fragmented and they fell further behind with the observed
> send queue overflow on the clients. We'd previously seen something
> similar with large HTTP streaming uploads.
>
> GRO can be turned off using:ethtool -K <NIC> gro off
>
> Cheers,
> Brendan
>
> (It's not the flow control, I know, I know ..)
>
>
>
>
> On Fri, Oct 26, 2012 at 2:41 PM, Simon MacMullen <simon at rabbitmq.com
> <mailto:simon at rabbitmq.com>> wrote:
>
>     On 26/10/12 11:23, Brendan Hay wrote:
>
>         Yes, so what should the expected observation be - the client code
>         carries on publishing into a black hole,
>         which means the send queue on the client/peer socket should keep
>         growing,
>
>
>     Well, the send queue is limited in size. So the publisher should
>     block fairly quickly.
>
>
>         and when the rabbit reader is
>         issued new credits, it will lap it all up?
>
>
>     Yes.
>
>
>         It just seems the 'flow' status in the UI stays on permanently, it
>         doesn't seem to be toggling at high speed, just locked.
>
>
>     Well, the flow status *in the UI* is designed not to toggle at high
>     speed, since that would not be very readable - it shows "flow" if
>     the connection has been blocked in the last 5 seconds. This is
>     driven by "last_blocked_age" and "last_blocked_by".
>
>     The rabbitmqctl command results you posted show some connections
>     which had blocked some time in the past, all more than 5 seconds
>     ago. Unfortunately I forgot to ask you to add "state" to the list of
>     columns, to determine if they were blocked now. If we see any
>     connections that were blocked by flow control a long time ago, and
>     are still blocked, then I'm concerned.
>
>
>     Cheers, Simon
>
>     --
>     Simon MacMullen
>     RabbitMQ, VMware
>
>


-- 
Simon MacMullen
RabbitMQ, VMware