[rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

[Ulli Berthold - Exactag GmbH]

Hi Matthias,

yes, we are experiencing major delays at the client in some (non-reproductible) cases.
So if I understand this correctly there is no predefined event to catch delays like that with the default client, instead I have to create a custom way to react on delays and flow limits? For example using the "internal" queuing you proposed?

With kind regards

Ulli

-----Ursprüngliche Nachricht-----
Von: Matthias Radestock [mailto:matthias at rabbitmq.com] 
Gesendet: Mittwoch, 8. August 2012 14:30
An: Ulli Berthold - Exactag GmbH
Cc: Discussions about RabbitMQ
Betreff: Re: AW: AW: AW: AW: AW: [rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

Ulli,

On 08/08/12 12:55, Ulli Berthold - Exactag GmbH wrote:
> well, it does happen here... There are times when our queues grow to a 
> couple of million messages (which actually was not a problem in 
> 2.7.1), in 2.8.4 it went into flow control mode when the CPU load got 
> higher though not limiting out.

When the management plug-in says that a connection is in 'flow' mode, that does *not* mean that it is no longer accepting messages. It only indicates that in the last ~10 seconds flow control got triggered at least once.

Also, when you say the CPU load wasn't limiting out, bear in mind that on a multi-core machine rabbit will not necessarily be able to utilise all cores - depends very much on the workload.

Are you actually experiencing publishing delays at the client?

> Our problem is that we constantly produce lots of messages which have 
> to be forwarded within milliseconds or we will run into frontend 
> delays which should not occur.

Yes, I understand that.

> I'm looking for a way to keep the queueing server accepting messages 
> at the highest possible rate no matter what, for as long as it's not 
> "full" (memory, disk, cpu).

One way to increase the buffering capacity is to increase the tcp buffer sizes in the kernel.

> And when we finally reach a point where the server is almost full, I'd 
> curious if the flow control events of the C# driver will ever be 
> fired? As I said, that would enable us to use a file-based fallback 
> that's mostly intended for queuing server downtimes.
>
> channel = connection.CreateModel(); channel.ChannelFlow(true); 
> channel.FlowControl += new 
> RabbitMQ.Client.Events.FlowControlEventHandler(channel_FlowControl);

RabbitMQ does not send channel.flow commands, so no, these events won't fire.

We were sending channel.flow from rabbit a few years ago but had to abandon that because a) it proved to be way too slow to react to critical server conditions, and b) many clients weren't handling channel.flow correctly.


I still think my client-side mini queue idea would work well for you. It provides a high degree of control over when to trigger a fallback publishing path and has the added advantage that it will also cope with temporary network disruptions - i.e. conditions that stall (but not
kill) tcp connections - which no server-generated signal could do.


Regards,

Matthias.