[rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

[Ulli Berthold - Exactag GmbH]

Hi Matthias (german name?)

My problem is that we are talking web tracking here, so it's at least 500 events per second, originating from 4 different web servers running 8 threads each on average. Data stream will increase to over 3.000/s as soon as we can handle that load. The basic understanding so far was that rabbit would cache the message queue on disk when there is too much incoming data, so we could use the quieter night times to handle the data that didn't make it in time.

I didn't see an easy way to check the queue length, except maybe for rest requests?

With kind regards

Ulli

-----Ursprüngliche Nachricht-----
Von: Matthias Radestock [mailto:matthias at rabbitmq.com] 
Gesendet: Mittwoch, 8. August 2012 09:58
An: Ulli Berthold - Exactag GmbH
Cc: Discussions about RabbitMQ
Betreff: Re: AW: [rabbitmq-discuss] TCP Backpressure / Flow Control in C# producer

Ulli,

On 08/08/12 08:51, Ulli Berthold - Exactag GmbH wrote:
> Well, because i got a producer that frequently puts more data into the 
> queue than is processed at that time (will processed be at later, 
> quieter, intervals though). Frequently that puts the producer into 
> flow mode, slowing it down, which is not an option in our case. The 
> producer has to be freed again within milliseconds to accept the next 
> event. If the queue doesn't accept data as fast as I produce it I need 
> to see that and switch over to an alternative (local file-based
> cache) which again will be processed later.

I suggest you introduce a second thread and connect that to the producer thread via a one-place buffer or a queue data structure. The second thread pulls items off that queue and publishes them to RabbitMQ. The publisher thread can look at the queue's length before deciding whether to place another item in it or write to disk.

Matthias.