[rabbitmq-discuss] Form Submission - Add A Question

[Sean Treadway]

Ben,

On Nov 22, 2008, at 00: 54, Ben Hood wrote:
>> I don't believe I can control any socket options with the library  
>> I'm using.
>
> Exactly my point. If you don't what your client is doing under the
> covers, you are not in a good position to reason with the behavior of
> the broker :-(

If I could control the consumer's channel's socket to minimize the  
number of mid-flight messages, what socket options should I set?

>> The test case that I'm basing my observations on is zipped up with
>> instructions in the README.  The tests require the EventMachine  
>> gem, and
>> amqp 0.5.9 or source to be installed.
>
> OK, I'll try to have a look at this.

Matthias answered the question that this test case covered, so there's  
no need to check the code here.

I understand better how the socket, channel and consumer behave in  
terms of what I'm thinking of as a "client".  Quite a bit more  
involved than a simple FIFO buffer ;)

I had assumed that since these mid-flight, un-acked messages would  
still exist in the broker, ready to be retried or persisted if they  
were published with delivery-mode=2, they should should be counted  
towards the 'message count'.  I assumed that only when the persistence  
rules would remove the message from the queue the message count would  
also decrement.

This makes it clearer: http://lettuce.squarespace.com/faq/queues/when-declaring-a-queue-what-does-the-message-count-field-mea.html 
   But on a practical level, I think it's important to remind  
explicitly that mid-flight messages for a consumer with no_ack=false  
are not included in this count.  And the maximum number of messages  
sent to a consumer is determined by the basic.qos branch which is  
currently pending QA, so at the moment, it's only limited by the  
number of bytes buffered by the transport layer.

On small scale testing (0-10 messages like I was using) it wasn't  
obvious that the message count was 0 because the 10 published, un- 
acked messages lived in my consumer's socket buffer.  Whereas for  
larger tests, where the number of mid-flight messages are a smaller  
proportion of the total messages, this statistic becomes more obvious  
(and useful).

One our use cases, parallelizing long running jobs, emphasizes  
reliability and accountability over throughput, so we would like to  
measure all unacknowledged messages in a queue or limit the number of  
messages consumed to one at a time.  Our goal is to keep the queue  
depth around 0-20 un-ack'd messages by controlling the number  
consumers..

It looks like branch 19684 (rabbitmqctl list_queues messages_ready)  
gives us the statistic we can use to tune our consumer pools.  Are  
there any plans of also exposing the 'messages_ready' statistic over  
AMQP?

Or would branch 18577 (basic.qos) with pre-fetch set to 1 give us the  
count of un-acknowledged messages we're looking for from a passive  
queue.declare?

>> I'm sure that Basic.Get can be used, but in my rough tests, I  
>> couldn't get
>> un-ack'd messages redelivered.  I believe I was failing to close my  
>> channel
>> to get them redelivered.  I'll also take another look at Basic.Get  
>> with
>> no_ack=true.
>
> http://lettuce.squarespace.com/faq/flow-control/how-long-does-it-take-to-redeliver-an-un-acknowledged-messag.html

:) A question about this answer... When you say "this can be done  
_manually_ with basic.get", does this imply that the redelivered  
message will be prioritized to the channel that issued the basic.get  
over a channel with a consumer?

Thanks,
Sean