[rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

Wed Aug 8 13:29:45 BST 2012

Ulli,

On 08/08/12 12:55, Ulli Berthold - Exactag GmbH wrote:
> well, it does happen here... There are times when our queues grow to
> a couple of million messages (which actually was not a problem in
> 2.7.1), in 2.8.4 it went into flow control mode when the CPU load got
> higher though not limiting out.

When the management plug-in says that a connection is in 'flow' mode, 
that does *not* mean that it is no longer accepting messages. It only 
indicates that in the last ~10 seconds flow control got triggered at 
least once.

Also, when you say the CPU load wasn't limiting out, bear in mind that 
on a multi-core machine rabbit will not necessarily be able to utilise 
all cores - depends very much on the workload.

Are you actually experiencing publishing delays at the client?

> Our problem is that we constantly produce lots of messages which have
> to be forwarded within milliseconds or we will run into frontend
> delays which should not occur.

Yes, I understand that.

> I'm looking for a way to keep the queueing server accepting messages
> at the highest possible rate no matter what, for as long as it's not
> "full" (memory, disk, cpu).

One way to increase the buffering capacity is to increase the tcp buffer 
sizes in the kernel.

> And when we finally reach a point where the server is almost full,
> I'd curious if the flow control events of the C# driver will ever be
> fired? As I said, that would enable us to use a file-based fallback
> that's mostly intended for queuing server downtimes.
>
> channel = connection.CreateModel(); channel.ChannelFlow(true);
> channel.FlowControl += new
> RabbitMQ.Client.Events.FlowControlEventHandler(channel_FlowControl);

RabbitMQ does not send channel.flow commands, so no, these events won't 
fire.

We were sending channel.flow from rabbit a few years ago but had to 
abandon that because a) it proved to be way too slow to react to 
critical server conditions, and b) many clients weren't handling 
channel.flow correctly.

I still think my client-side mini queue idea would work well for you. It 
provides a high degree of control over when to trigger a fallback 
publishing path and has the added advantage that it will also cope with 
temporary network disruptions - i.e. conditions that stall (but not 
kill) tcp connections - which no server-generated signal could do.

Regards,

Matthias.