[rabbitmq-discuss] Per-Connection Flow Control -- The Case Against (Try #2)

Wed May 30 06:41:23 BST 2012

It would appear Google Groups ate my reply; thus, this resend.  Sorry if
this is a dup for anyone.
--------------------

 On 5/29/2012 6:21 AM, Tony Garnock-Jones wrote:

On 28 May 2012 19:12, Reverend Chip <rev.chip at gmail.com> wrote:

> That's very funny.
>  If you're serious, then I can't take you seriously.
>

 I suspect you're just being rude....

I explained a scenario, as produced by the current flow control code, that
is ridiculous on its face: That growing queues are fine, until consumers
appear, at which point growing queues are not fine.  That's a paradoxical
result that at the very least should prompt reevaluation of the flow
control algorithm.  To ask what the practical bad results are in my
specific case is ridiculous misdirection.

Either growing queues (up to disk and memory limits) are acceptable -- my
position, and the position of RMQ without consumers -- or growing queues
are not acceptable -- the position of RMQ with consumers.  This paradox is
quite simple to write and to read.  The behavior of the current code is
absurd, as anyone can see if they were to choose to look.

So no, I was not being rude, specifically.  But I did ridicule your
comment, which well deserved it.  Per Thomas Jefferson: "Ridicule is the
only weapon which can be used against unintelligible propositions. Ideas
must be distinct before reason can act upon them."

PS: You may not have thought about the further wrinkle that the queues do
not accept new messages until they are EMPTY.  Which is doubly ridiculous.

 , but since it's also possible that the message you're replying to didn't
get through to you in its entirety (because of broken "From" quoting at
some stage in the mail delivery pipeline), I'll reproduce the rest of what
I wrote here in case it helps advance the discussion:

 >From one point of view, any time consumers exist but are not keeping up
with offered traffic is a good time to throttle the producers. Perhaps call
this the "steady state" point of view - I'm unsure if it has a better name
already.

 >From another point of view - call this the "bursty" point of view,
perhaps - the growth of queues is perfectly acceptable, whether consumers
exist at the time or not.

 ISTM Rabbit makes a guess about whether the application is "steady state"
or "bursty", choosing "steady state" all the time. Rabbit team, is this the
case? Is there a way of configuring the flow control to permit unbounded
queue growth to support the "bursty" case? Or do I have the wrong end of
the stick entirely?

> 1. A client is publishing 1.5K/sec to each of four exchanges, each of
> which has a queue.
> 2. There are no consumers.  Therefore the queue is growing.
> 3. RMQ does not stop this.  "Flow control" does not trigger.
> 4. The consumers appear to begin to tear down the backlog.
>  5. RMQ per-connection flow control suddenly decides that now there are
> some consumers, now it has a reason to throttle the sender.
> 6. If the consumers had not shown up, the producer would not have been
> blocked.
>

 7. The consumers drain some of the messages from the queues.
8. The producers are unblocked.
9. Profit!!!

 Regards,
  Tony
-- 
Tony Garnock-Jones
tonygarnockjones at gmail.com
http://homepages.kcbbs.gen.nz/tonyg/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120529/3cc5908b/attachment.htm>