[rabbitmq-discuss] Producer confirms as a heuristic for rate limiting

[Matthias Radestock]

Ben,

On 28/02/13 08:49, Ben Hood wrote:
> On Wednesday, 27 February 2013 at 18:52, Matthias Radestock wrote:
>
>> Using confirms to exert back pressure on a producing application is
>> fine. But I wouldn't expect that to result in particularly stable
>> latencies. For a few reasons:
>>
>> 1) one way that the internal confirm logic gets triggered is by "absence
>> of other stuff happening", so when throttling producers, confirms may
>> well happen a lot quicker.
>>
>
> If I understand you correctly, then this is precisely the feedback loop
> that I'm trying to achieve - when the sender is throttled, this gives
> the internals of Rabbit more head room to do whatever it is a Rabbit
> does. In Rabbit becoming more free, the rate of confirms increases, and
> hence the more credit the sender will receive. When the sender ramps up
> again, the reverse should happen, and so the feedback cycle continues.

My point is that the feedback signal is highly unstable, and the 
correlation between message send rate and confirm roundtrip time is 
highly non-linear. So you'd probably have to do a fair amount of 
processing on the signal in order to make the feedback smooth.

>> 2) more generally, much of the internal confirm logic will try to batch
>> things if it can, which reduces the overall cost but makes the timing of
>> confirms more variable.
>
> That's understandable - fine grained ingress credits would come at a
> premium in any case.
>
> In my scenario I'm setting a timestamp into the message header and the
> receiver is decoding this in order to compare it with its current
> timestamp. With the measurement being end to end, would the asynchronous
> nature of the confirms not be obviated by the fact that you are
> measuring the speed of the uplink plus the routing overhead plus the
> speed of a separate delivery (he says had waving)?

My point here is the same as the previous one, i.e. the flow of confirms 
is anything but smooth, which makes it tricky to construct feedback 
loops based on that which result in stable end-to-end latencies.

>> 3) confirms do come at a noticeable penalty on throughput and latency.
>> So a benchmark employing confirms would not be a very good proxy for a
>> real system running w/o confirms.
>
> What constitutes the performance penalty in this case? Do confirms
> increase the latency because they add to the measurable round trip as
> seen by the sender, or does the fact that they are turned on add extra
> processing effort which is CPU time that has to be spread over each
> message and hence on average adding latency to each message? If the
> answer is the latter, does the same apply to throughput - since confirms
> have a non-zero accounting cost, having them turned on diminishes the
> ability to process as many items in the same amount of time?

All the latter. i.e. confirms take some CPU time. which due to the 
batching and various other optimisations is not spread evenly across 
messages.

>> Having said all that, go and try it :)
>
> I'm very tempted, but the reason why I'm asking first is to avoid going
> down any rabbit holes that a more experienced person could potentially
> advise me to avoid :-)

Well, MulticastMain can already throttle producers based on a "max 
number of outstanding confirms". So playing with that would be a 
starting point.

Matthias.