[rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

[Ulli Berthold - Exactag GmbH]

Matthias,

Rabbit does handle it, yes, but it slows down the inbound messages at some point (like several million messages in queue, with lots of disk space still available).

And we are using rabbit so we don't have to implement our very own queueing system which would likely just add much more complexity, again, we're talking millions of messages per hour. Just a single minute of backlog in the local queue means 30.000+ messages stored locally... We considered RabbitMQ / Shovel but I'm not too sure that would keep up with the load.

We would love a way to disable (or better, suppress) the whole backpressure system until the server is really at its memory and disk limit. Currently it seems to partially depend on the processing speed of the queue readers and the queue sizes. Again, an alternative to just know when backpressure is applied would help us too because we could temporarily send the next messages through an alternate route. An in-program queue would definitely increase complexity almost beyond the point of usefulness of the queueing system - why use an existing one if I got to build one anyways?

Regards

Ulli

-----Ursprüngliche Nachricht-----
Von: Matthias Radestock [mailto:matthias at rabbitmq.com] 
Gesendet: Mittwoch, 8. August 2012 10:21
An: Ulli Berthold - Exactag GmbH
Cc: Discussions about RabbitMQ
Betreff: Re: AW: AW: [rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

Ulli,

On 08/08/12 09:03, Ulli Berthold - Exactag GmbH wrote:
> My problem is that we are talking web tracking here, so it's at least
> 500 events per second, originating from 4 different web servers 
> running 8 threads each on average. Data stream will increase to over 
> 3.000/s as soon as we can handle that load.

RabbitMQ should be able to handle that easily, even when writing to disk.

> The basic understanding
> so far was that rabbit would cache the message queue on disk when 
> there is too much incoming data, so we could use the quieter night 
> times to handle the data that didn't make it in time.

Right.

> I didn't see an easy way to check the queue length, except maybe for 
> rest requests?

I wasn't talking about an AMQP queue. I suggested that at the client you have the following setup:

publisher thread -> .net queue -> shoveling thread

The helper thread pulls items off the .net queue and publishes them to RabbitMQ. The publisher thread can look at the queue's length before deciding whether to place another item in it or write to disk. 
Effectively the length of the .net queue becomes a measure of the TCP back pressure experienced by the shoveling thread.

Matthias.