[rabbitmq-discuss] Fwd: Batch Acknowledge

[Davide Maestroni]

BTW, does it make sense to use transactions in the consumer?

Davide

---------- Forwarded message ----------
From: Davide Maestroni <davide.maestroni at gmail.com>
Date: Wed, Oct 20, 2010 at 2:58 PM
Subject: Re: [rabbitmq-discuss] Batch Acknowledge
To: Simon MacMullen <simon at rabbitmq.com>


Hi Simon,

thank you very much for the quick reply!
Now I have a better understanding on how the message flow works. Still I
cannot find a way to achieve correct message ordering even after a crash.
My point here is: as soon as I read a message from the queue, the order of
that message is virtually lost, even if I try immediately to write the
message somewhere else. So if a crash happens after reading something,
either through "basic.get" or "basic.consume", it does not matter if the
acknowledge was sent or not, in any case I have no way to get the same
message(s) as the first one(s) after the recovery.
Is there no method to just "peek" into the queue? or, can you suggest me a
way to ensure ordering AND not lose any message at the same time?

Thanks,

Davide


On Wed, Oct 20, 2010 at 12:15 PM, Simon MacMullen <simon at rabbitmq.com>wrote:

> On 19/10/10 18:30, Davide Maestroni wrote:
>
>> What I want to achieve is a queue in which all the messages persist
>> until a "consumer" (not sure if it is really a consumer actually)  do
>> some batch processing on N messages, like updating a separate DB, and,
>> only when everything has completed correctly, send an acknowledge so
>> that the queue can release the last delivered N messages.
>> As per my understanding I can "declare" a queue and then start reading
>> from it using basicGet() API. Then, after processing the desired number
>> of messages, I call basicAck(deliveryTag, true) passing the tag of the
>> last read message.
>> In my case it is both important to keep the message ordering and not to
>> lose any message even if the consumer or the broker crash.
>>
>
> Note that we only guarantee message ordering when messages are not
> redelivered. If your client dies with unacked messages about, those messages
> will be reinserted into the queue.
>
> Example: the queue contains messages ABCDEF. You client consumes ABC, does
> not ack them, and dies. The queue will now contain DEFABC.
>
>
>  In order to
>> achieve that, I am going to publish messages "mandatory" and
>> "persistent" and then send an ack in the consumer after successfully
>> securing the data in some other place.
>> What I want to be clear here is: is it correct to say that if the
>> consumer crashes after M basicGet() (but before sending the ack), at
>> restart the same M messages will be returned calling basicGet()? I.e.
>> are the messages removed from the queue only after receiving the
>> acknowledge?
>>
>
> They are only finally removed when the server sees the ack, but see above
> about ordering.
>
>
>  If it is not so, do I have to call basicRecover()?
>>
>
> No. basic.recover allows you to say "pretend I crashed". It's not 100%
> obvious why this is useful, but it's in the spec. Note that you don't have
> to call recover after recovering from crashing...
>
>
>  Can I do it even if I don't call basicConsume()?
>>
>
> Yes, the rules are exactly the same for consume vs get.
>
>
>  And most importantly, which is the order of
>> the messages when calling basicRecover() after a crash and restart of
>> the consumer node?
>>
>
> Hopefully explained above.
>
>
>  A final question: if a message is flagged as "mandatory", how long the
>> queue wait for a consumer to be ready before returning the message to
>> the sender?
>>
>
> "Mandatory" just means the message must get routed to a queue, not that
> anything must be listening on the queue. I think you mean "immediate". And
> in the immediate case, it really is immediate; there must be a consumer on
> the queue (no basic.get!), no messages already in the queue and the consumer
> must not be blocked for basic.qos or channel.flow reasons.
>
> Cheers, Simon
> --
> Simon MacMullen
> Staff Engineer, RabbitMQ
> SpringSource, a division of VMware
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101020/e9a69641/attachment-0001.htm>