[rabbitmq-discuss] How to do blocking asynchronously with Pika? I want 1 message at a time and stop receiving until I'm ready again.

[Matthias Radestock]

Scott,

On 21/02/13 18:11, Scott Chapman wrote:
> If 1st message arrives, I ack it, and I am processing it then 2nd message
> arrives, it will be held _in_the_client_?

Yes.

> So if the client gets rebooted, the message will be lost?

No. The server holds on to messages until they have been acknowledged.

You may want to read 
http://www.rabbitmq.com/tutorials/tutorial-two-python.html, if you 
haven't done so already.

> It appears that I can set the basic.qos prefetch-count to zero to solve
> this problem?

No. Zero means "unlimited". But there is no problem to start with.

> If I decide to cancel the consumer,  I assume the proper order of events
> would be:
>
> Enable Consumer
> Receive message 1
> Cancel Consumer
> Ack Message
> Process Message
> -> back to Enable Consumer.

What do you want to happen when the client crashes during Process Message?

The typical consumer sequence, in AMQP commands, is

- basic.qos{prefetch=n}
- basic.consume
- on basic.deliver (i.e. receiving of a message)
   - process message
   - basic.ack

If the prefetch is set to 1 (though see note below), we can guarantee 
that the client only ever has at most one message sent to it until it 
issues an ack. And sending the ack after processing ensures that if 
anything goes wrong during processing, e.g. the client crashes, then the 
server will requeue the message, so other clients (or the same client 
when it reconnects) can process it.

In some circumstances you may want to do the following instead:

- basic.qos{prefetch=n}
- basic.consume
- on basic.deliver
   - basic.cancel
   - process message until a safe point from which the client can recover
   - basic.ack
   - do more, possibly expensive, processing of the message
   - basic.consume

which is closer to what you envisaged. But that's a lot more complex and 
expensive, so only warranted if the workload fits that special pattern 
of message processing being very expensive and splittable into two phases.

> I think one would not want to ACK the message before canceling the
> consumer because it might immediately dump another message into the
> consumer?

Correct.

Now back to setting the prefetch. For maximum performance you'd want to 
tune that s.t. whenever the client has finished processing a message 
then the next message has just arrived from the server moments earlier. 
That way the client never sits there idle waiting for a message or, 
conversely, buffers a whole bunch of messages that could be processed by 
other consumers. The right figure here is highly dependent on workload 
and network latency.


Regards,

Matthias.