[rabbitmq-discuss] Per-Connection Flow Control - RMQ 2.8.1

[Matthias Radestock]

On 27/04/12 15:42, DawgTool wrote:
> On 4/25/12 12:36 PM, Matthias Radestock wrote:
>> 2.7.1 can absorb longer spikes than 2.8.1; you can compensate for that
>> somewhat by increasing the tcp buffer sizes.
>>
> Agreed that 2.7.1 absorbs the spikes, which I think a message broker
> should be able to do.

Both 2.7.1 and 2.8.1 can absorb spikes. 2.7.1 can absorb longer ones by 
default but at a price (see below).

> Accept the message, route when message is ready or queue is available.

The downside of this approach is that the broker builds up an enormous 
backlog of work. Eventually it is going to have to catch up. And when it 
does then message acceptance rate slows to a crawl, or indeed stops 
completely for sustained periods of time. That is undesirable in most 
use cases.

Furthermore, the per-message cost of processing messages that way is 
higher since the system is more highly stressed - memory allocation, 
garbage collection, scheduling all become more expensive. And Rabbit's 
page-to-disk logic is working sub-optimally since it operates only on 
messages residing in queues. All this combined results in the number of 
messages that the broker can process over a longer period of time to be 
noticeably lower than with the new credit flow control.

Having said that, the ultimate "accept messages as quickly as you can; 
think about them later" behaviour is achieved by increasing the tcp 
buffer sizes, as I noted in my previous reply. So if that's the 
behaviour you are after, then I'd suggest you focus your tuning efforts 
there.

Matthias.