[rabbitmq-discuss] Per-Connection Flow Control -- The Case Against

[Reverend Chip]

On 5/30/2012 3:56 AM, Simon MacMullen wrote:
> On 30/05/12 06:52, Chip Salzenberg wrote:
>> On Fri, May 25, 2012 at 5:12 AM, Simon MacMullen <simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>> wrote:
>>
>>     In the mean time, I wonder whether 500 consumers and prefetch-count
>>     of 2 is what you really want. Normally I would expect a
>>     configuration like that when you expect to take some time to process
>>     each message, but it sounds like you're going fast enough that
>>     either 500 consumers are not needed, or prefetch-count > 2 might
>>     lead to better performance.
>>
>>
>> I didn't know that a larger prefetch count could reduce broker workload.
>>   Per-message processing in this system is sometimes high (up to 20s),
>> but it averages very low.  It's likely that 500 is overkill, but
>> probably only by 2x or so.
>>
>> Assuming a constant throughput of about 6K/sec and at least 250 workers,
>> what prefetch seems more helpful, IYO?  10?  20?
>
> Well, there are two different things here that I somewhat conflated:
>
> * Number of consumers using prefetch count
>   - The higher this number, the more bookkeeping work the queue has
>     to do.
>
> * Size of prefetch count
>   - The lower this number, the more likely that any given consumer will
>     stall, and thus the slower consuming goes.

Ah.  Well, throughput is not our problem, flow control on publish is. 
So the latter issue is off the table.  The former issue should be
negligible; what modern software has trouble tracking 500 queue depths?

Meanwhile, consider this.  In our system, when stalls happened so
uselessly, actually had 2 queues bound to the exchange.  And it was
definitely the flow control kicking in and not the queue becoming
overwhelmed because a) it said "flow" in the ui. b) both queues were
blocked, not just the one that had tons of consumers starting on it. The
flow control is failing worse than I had already described because if
prevents other queues bound to the same exchange from getting any
messages even if they are empty.