[rabbitmq-discuss] Form Submission - Add A Question

[Ben Hood]

Sean,

On Mon, Nov 24, 2008 at 1:09 PM, Sean Treadway <sean at soundcloud.com> wrote:
> Ben,
>
> On Nov 22, 2008, at 00: 54, Ben Hood wrote:
> If I could control the consumer's channel's socket to minimize the number of
> mid-flight messages, what socket options should I set?

I think you've understood me out of context - this is what I originally said:

>> This will depend in part on how your client behaves, e.g. whether it
>> absorbs socket back pressure or not.

What I meant by absorbing back pressure would be an example where the
consuming client pulls every message from the socket and buffers it
internally. This would negate the effect of back pressure and just
exercise memory pressure on the client application to compensate for
this. This would an example where you would be throwing away the flow
control provided by TCP (under the assumption that you are using TCP
as a transport).

> This makes it clearer:
> http://lettuce.squarespace.com/faq/queues/when-declaring-a-queue-what-does-the-message-count-field-mea.html
> But on a practical level, I think it's important to remind explicitly that
> mid-flight messages for a consumer with no_ack=false are not included in
> this count.

Sorry, I don't quite follow this. Can you explain?

> One our use cases, parallelizing long running jobs, emphasizes reliability
> and accountability over throughput, so we would like to measure all
> unacknowledged messages in a queue or limit the number of messages consumed
> to one at a time.  Our goal is to keep the queue depth around 0-20 un-ack'd
> messages by controlling the number consumers..

This sounds more like an application of basic.qos because you would to
distribute messages based on the various consumers' ability to process
units of work. By setting the prefetch window, you get more fine
grained consumer flow control.

> It looks like branch 19684 (rabbitmqctl list_queues messages_ready) gives us
> the statistic we can use to tune our consumer pools.  Are there any plans of
> also exposing the 'messages_ready' statistic over AMQP?

Hmmm, difficult question. Some of the Redhat guys have been talking
about putting more managementy stuff into the protocol, but I don't
know what the current situation.

And coincidentally we have talking about using the codegen to add some
propriety methods to Rabbit, but this is just talk.

There were further discussions about SNMP as well.

I think for now now backing an embedded RPC handler using JSON as a
codec will pay even money. This would be quite simple to knock up with
the Erlang client for example.

> Or would branch 18577 (basic.qos) with pre-fetch set to 1 give us the count
> of un-acknowledged messages we're looking for from a passive queue.declare?

I'm not sure I understand - if you set the prefetch size to be one,
only one outstanding message per channel is allowed - I don't think
this answers your question though.

>> http://lettuce.squarespace.com/faq/flow-control/how-long-does-it-take-to-redeliver-an-un-acknowledged-messag.html
>
> :) A question about this answer... When you say "this can be done _manually_
> with basic.get", does this imply that the redelivered message will be
> prioritized to the channel that issued the basic.get over a channel with a
> consumer?

No, this just a way to quickly grab that one message that wasn't
acked. It would make no difference if you were to start another
consumer for the same queue on a different channel - this is 6 of one
and half a dozen of the other.

HTH,

Ben