[rabbitmq-discuss] Fwd: Batch Acknowledge

Wed Oct 20 16:23:50 BST 2010

Hi Simon,

thank you again. I see that transactions are only meant for publications, so
they have no use in my case.
My problem here is that, in the architecture I have to implement, I could
receive a lot of messages in a very short time, so if the broker re-enqueues
not acknowledged ones they can end up to be re-sent thousands of messages
later. Which means that there's no way I can do the re-ordering on the
consumer side.
The first idea that comes into my mind is to re-publish the consumed
messages in a separate dedicated queue, so to limit the number of messages
in it, and then stop reading from the first queue until all the messages in
the second one has been read, re-ordered and correctly processed. Though,
there's still a case in which the consumer crashes before being able to
re-enqueue one message, thus effectively losing it.
I really can't see a solution to my problem... :-(

Best,

Davide

On Wed, Oct 20, 2010 at 5:00 PM, Simon MacMullen <simon at rabbitmq.com> wrote:

> For your case it doesn't make much difference. The usual use case for
> consuming with transactions is some sort of message transformer that needs
> to consume and publish atomically.
>
> Cheers, Simon
>
>
> On 20/10/10 15:56, Davide Maestroni wrote:
>
>> BTW, does it make sense to use transactions in the consumer?
>>
>> Davide
>>
>> ---------- Forwarded message ----------
>> From: *Davide Maestroni* <davide.maestroni at gmail.com
>> <mailto:davide.maestroni at gmail.com>>
>> Date: Wed, Oct 20, 2010 at 2:58 PM
>> Subject: Re: [rabbitmq-discuss] Batch Acknowledge
>> To: Simon MacMullen <simon at rabbitmq.com <mailto:simon at rabbitmq.com>>
>>
>>
>> Hi Simon,
>>
>> thank you very much for the quick reply!
>> Now I have a better understanding on how the message flow works. Still I
>> cannot find a way to achieve correct message ordering even after a crash.
>> My point here is: as soon as I read a message from the queue, the order
>> of that message is virtually lost, even if I try immediately to write
>> the message somewhere else. So if a crash happens after reading
>> something, either through "basic.get" or "basic.consume", it does not
>> matter if the acknowledge was sent or not, in any case I have no way to
>> get the same message(s) as the first one(s) after the recovery.
>> Is there no method to just "peek" into the queue? or, can you suggest me
>> a way to ensure ordering AND not lose any message at the same time?
>>
>> Thanks,
>>
>> Davide
>>
>>
>> On Wed, Oct 20, 2010 at 12:15 PM, Simon MacMullen <simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>> wrote:
>>
>>    On 19/10/10 18:30, Davide Maestroni wrote:
>>
>>        What I want to achieve is a queue in which all the messages persist
>>        until a "consumer" (not sure if it is really a consumer
>>        actually)  do
>>        some batch processing on N messages, like updating a separate
>>        DB, and,
>>        only when everything has completed correctly, send an acknowledge
>> so
>>        that the queue can release the last delivered N messages.
>>        As per my understanding I can "declare" a queue and then start
>>        reading
>>        from it using basicGet() API. Then, after processing the desired
>>        number
>>        of messages, I call basicAck(deliveryTag, true) passing the tag
>>        of the
>>        last read message.
>>        In my case it is both important to keep the message ordering and
>>        not to
>>        lose any message even if the consumer or the broker crash.
>>
>>
>>    Note that we only guarantee message ordering when messages are not
>>    redelivered. If your client dies with unacked messages about, those
>>    messages will be reinserted into the queue.
>>
>>    Example: the queue contains messages ABCDEF. You client consumes
>>    ABC, does not ack them, and dies. The queue will now contain DEFABC.
>>
>>
>>        In order to
>>        achieve that, I am going to publish messages "mandatory" and
>>        "persistent" and then send an ack in the consumer after
>> successfully
>>        securing the data in some other place.
>>        What I want to be clear here is: is it correct to say that if the
>>        consumer crashes after M basicGet() (but before sending the ack),
>> at
>>        restart the same M messages will be returned calling basicGet()?
>>        I.e.
>>        are the messages removed from the queue only after receiving the
>>        acknowledge?
>>
>>
>>    They are only finally removed when the server sees the ack, but see
>>    above about ordering.
>>
>>
>>        If it is not so, do I have to call basicRecover()?
>>
>>
>>    No. basic.recover allows you to say "pretend I crashed". It's not
>>    100% obvious why this is useful, but it's in the spec. Note that you
>>    don't have to call recover after recovering from crashing...
>>
>>
>>        Can I do it even if I don't call basicConsume()?
>>
>>
>>    Yes, the rules are exactly the same for consume vs get.
>>
>>
>>        And most importantly, which is the order of
>>        the messages when calling basicRecover() after a crash and
>>        restart of
>>        the consumer node?
>>
>>
>>    Hopefully explained above.
>>
>>
>>        A final question: if a message is flagged as "mandatory", how
>>        long the
>>        queue wait for a consumer to be ready before returning the
>>        message to
>>        the sender?
>>
>>
>>    "Mandatory" just means the message must get routed to a queue, not
>>    that anything must be listening on the queue. I think you mean
>>    "immediate". And in the immediate case, it really is immediate;
>>    there must be a consumer on the queue (no basic.get!), no messages
>>    already in the queue and the consumer must not be blocked for
>>    basic.qos or channel.flow reasons.
>>
>>    Cheers, Simon
>>    --
>>    Simon MacMullen
>>    Staff Engineer, RabbitMQ
>>    SpringSource, a division of VMware
>>
>>    _______________________________________________
>>    rabbitmq-discuss mailing list
>>    rabbitmq-discuss at lists.rabbitmq.com
>>    <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>
>>    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
> --
> Simon MacMullen
> Staff Engineer, RabbitMQ
> SpringSource, a division of VMware
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101020/3e144434/attachment-0001.htm>