[rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

Matthias Radestock matthias at rabbitmq.com
Wed Aug 8 09:21:22 BST 2012


Ulli,

On 08/08/12 09:03, Ulli Berthold - Exactag GmbH wrote:
> My problem is that we are talking web tracking here, so it's at least
> 500 events per second, originating from 4 different web servers
> running 8 threads each on average. Data stream will increase to over
> 3.000/s as soon as we can handle that load.

RabbitMQ should be able to handle that easily, even when writing to disk.

> The basic understanding
> so far was that rabbit would cache the message queue on disk when
> there is too much incoming data, so we could use the quieter night
> times to handle the data that didn't make it in time.

Right.

> I didn't see an easy way to check the queue length, except maybe for
> rest requests?

I wasn't talking about an AMQP queue. I suggested that at the client you 
have the following setup:

publisher thread -> .net queue -> shoveling thread

The helper thread pulls items off the .net queue and publishes them to
RabbitMQ. The publisher thread can look at the queue's length before 
deciding whether to place another item in it or write to disk. 
Effectively the length of the .net queue becomes a measure of the TCP 
back pressure experienced by the shoveling thread.

Matthias.


More information about the rabbitmq-discuss mailing list