[rabbitmq-discuss] acks from temporary queues

[Alexandru Scvorţov]

Hi Jonathan,

> This is a point of confusion for me then. In a previous note where I
> mentioned setting autoAck to false and manually using basicAck,
> Matthias replied as follows:

Uf, I forgot about the persister in the last message.  Sorry.

> If in the fanout context I could have used consumer acks to hold off
> broker acks to the publisher, I would have expected that I could
> effectively block confirms until after I sent basicAcks from the
> consumer to the broker and then on up to the publisher. From
> Matthias's note (and my tests), however, it appears that the broker
> sends an ack back to the publisher as soon as it takes control of the
> message and so issues a confirm without waiting on anything from the
> consumer. 

He's right.  I was hoping the immediate flag will hold off the confirm,
but, of course, it won't.

> In the direct exchange context, however, I took your notes
> to be indicating that the broker there will wait for an ack from the
> consumer, such that if the consumer died while in process, but before
> sending the ack, the broker would return a basic.return along with the
> confirm. If this is not the case, then, my sense is that I will have
> to have the consumers publish a response directly back to original
> publisher as Matthias notes--while using the fanout broadcast to keep
> track of the number of expected responses (as you indicate).

You will have to have consumers send something to the original
publisher and a fanout to keep track of responses.  You'll only need a
timeout of some sort, so that if any of the consumers takes to long to
respond, you react somehow (that probably means the consumer died
processing the message).

You'll still need the mandatory, immediate flags to at least see that
the consumer was alive and was consuming from the queue at the time the
message was sent.

> Does that make sense? Basically, it still orbits around a question of
> how and/or if consumer acks can be made to effect a producer. I am
> guessing that my implementation will have to require the return
> publications (responses), but any clarification you could add to the
> consumer ack, confirm chain would be greatly appreciated.

You'll need to publish back.  The only things confirms will give you in
this case is the guarantee that basic.returns happen before confirms.
So, once you receive the confirm, you know the message has reached the
consumer.

There's also no need to delay acknowledgement on the consumer side.

Sorry for the confusion.

Alex


On Mon, Sep 26, 2011 at 09:04:06AM -0700, gneeri wrote:
> Hi Alex,
> 
> This is a point of confusion for me then. In a previous note where I
> mentioned setting autoAck to false and manually using basicAck,
> Matthias replied as follows:
> 
> "Confirms mean one thing and one thing only: that the broker has
> accepted
> responsibility for the message.
> 
> "They concern the interaction between producers and the broker, and
> have
> nothing to do with consumer acks."
> 
> If in the fanout context I could have used consumer acks to hold off
> broker acks to the publisher, I would have expected that I could
> effectively block confirms until after I sent basicAcks from the
> consumer to the broker and then on up to the publisher. From
> Matthias's note (and my tests), however, it appears that the broker
> sends an ack back to the publisher as soon as it takes control of the
> message and so issues a confirm without waiting on anything from the
> consumer. In the direct exchange context, however, I took your notes
> to be indicating that the broker there will wait for an ack from the
> consumer, such that if the consumer died while in process, but before
> sending the ack, the broker would return a basic.return along with the
> confirm. If this is not the case, then, my sense is that I will have
> to have the consumers publish a response directly back to original
> publisher as Matthias notes--while using the fanout broadcast to keep
> track of the number of expected responses (as you indicate).
> 
> Does that make sense? Basically, it still orbits around a question of
> how and/or if consumer acks can be made to effect a producer. I am
> guessing that my implementation will have to require the return
> publications (responses), but any clarification you could add to the
> consumer ack, confirm chain would be greatly appreciated.
> 
> Thanks,
> Jonathan
> 
> 
> On Sep 26, 11:34 am, Alexandru Scvorţov <alexan... at rabbitmq.com>
> wrote:
> > Hi Jonathan,
> >
> > > "In addition, when processing "please log out this user" messages, a
> > > machine should *not* acknowledge that it has received the message
> > > until after it publishes its response (this way, if the machine dies
> > > while processing, the sender will get a basic.return for it)."
> >
> > > Am I correct in understanding from this that, in the direct exchange
> > > context, I can control when the consumer machine sends back its ack,
> > > by calling something like basicAck at some point in the consumption/
> > > delivery code?
> >
> > The exchange type has nothing to do with it.  When you consume from a
> > queue, you specify the autoAck parameter.  If this is set to true
> > (which it is, by default), then the server will not wait for the
> > consumer to acknowledge receipt of the message (so, it's effectively
> > auto-ack'ing all messages).  If it's set to false, the broker will not
> > consider the message delivered to the consumer until it explicitly ack's
> > the message.  See the API guide for an example (I assume you're using
> > the Java client, but the other clients have very similar APIs):
> >  http://www.rabbitmq.com/api-guide.html#consuming
> >
> > > If so, I would expect that I could skip the return
> > > publishing (i.e., the response) from the consumer to the producer and
> > > simply check confirms and confirms + basic.returns for each message
> > > sent directly to each machine. Moreover, if, prior to publishing, I
> > > associate the machine (or direct queue corresponding to it) to the
> > > message to be sent, I should be able to find out which machines
> > > present which form of response (i.e., confirm or confirm +
> > > basic.return). Does this make sense or am I still reading too much
> > > into the consumer ack (such that, for example, the confirmation
> > > received by the publisher just represents the handoff of the message
> > > to the queue and not its uptake by the consumer)?
> >
> > Oh.  I missed that.  Yes, I think that would work.
> >
> > Cheers,
> > Alex
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 26, 2011 at 08:04:28AM -0700, gneeri wrote:
> > > Hi Alex,
> >
> > > Thanks so much for the detailed reply. I had begun thinking along
> > > these lines, but really appreciate the level of detail regarding the
> > > specific types of publish requests, queues and exchanges. This is
> > > really helpful and will likely save me many hours of trial and error.
> > > I do have just one quick question though regarding the following:
> >
> > > "In addition, when processing "please log out this user" messages, a
> > > machine should *not* acknowledge that it has received the message
> > > until after it publishes its response (this way, if the machine dies
> > > while processing, the sender will get a basic.return for it)."
> >
> > > Am I correct in understanding from this that, in the direct exchange
> > > context, I can control when the consumer machine sends back its ack,
> > > by calling something like basicAck at some point in the consumption/
> > > delivery code? If so, I would expect that I could skip the return
> > > publishing (i.e., the response) from the consumer to the producer and
> > > simply check confirms and confirms + basic.returns for each message
> > > sent directly to each machine. Moreover, if, prior to publishing, I
> > > associate the machine (or direct queue corresponding to it) to the
> > > message to be sent, I should be able to find out which machines
> > > present which form of response (i.e., confirm or confirm +
> > > basic.return). Does this make sense or am I still reading too much
> > > into the consumer ack (such that, for example, the confirmation
> > > received by the publisher just represents the handoff of the message
> > > to the queue and not its uptake by the consumer)?
> >
> > > Thanks again, really, I very much appreciate your help.
> >
> > > Cheers,
> > > Jonathan
> >
> > > On Sep 26, 5:54 am, Alexandru Scvorţov <alexan... at rabbitmq.com> wrote:
> > > > Hi Jonathan,
> >
> > > > > 1. Do consumer acks have more significance in transactions, such that
> > > > > I could use those to effect the functionality I'm looking for?
> >
> > > > Not really, no.  I don't think there's anyway to get what you want
> > > > automatically.
> >
> > > > > 2. If not, is there any way (apart from the admin HTTP API) for
> > > > > determining the number of consumers (e.g., bound queues) for a given
> > > > > exchange? This way I could test the number of reply messages as you
> > > > > suggest; otherwise, with needing to register and account for a dynamic
> > > > > array of servers, I would lose a good deal of what I had hoped would
> > > > > be the benefit of a fanout broadcast.
> >
> > > > There's no AMQP way of finding how many queues are bound to an exchange.
> > > > As you say, there's the management API, but using that is likely to lead
> > > > to races (what happens if you query the number of queues, then one of
> > > > the machines dies before it sends its acknowledgement?).
> >
> > > > You could kind of get what you want with AMQP.  Use a fanout exchange
> > > > like you want for broadcast messages, but use a direct exchange for
> > > > direct communication between machines.  You could use the fanout
> > > > exchange to make sure that all the machines know about each other (in
> > > > particular, that each knows the queues the others bound to the direct
> > > > exchange, so that they can communicate directly).  And you could use the
> > > > direct exchange to publish persistent, confirm, mandatory, immediate messages
> > > > directly to other machines.  With these messages, the sender will be
> > > > notified if the other machine received the message (only a confirm is
> > > > received for the publish), or if the other machine went offline in the past
> > > > (a basic.return and a confirm is received).  This way, the sending
> > > > machine would know which other machines received its message and, so,
> > > > will know to wait for the correct number of responses.  The queue bound
> > > > to the direct exchange would have to be durable, auto-delete for this to
> > > > work (durable for confirms to work and auto-delete so that once the sole
> > > > consumer dies, the queue is deleted and publishing to it will result in
> > > > a basic.return).  In addition, when processing "please log out this
> > > > user" messages, a machine should *not* acknowledge that it has
> > > > received the message until after it publishes its response (this way, if
> > > > the machine dies while processing, the sender will get a basic.return for
> > > > it).
> >
> > > > Hope this helps.
> >
> > > > Cheers,
> > > > Alex
> >
> > > > On Fri, Sep 23, 2011 at 04:44:14PM -0700, gneeri wrote:
> > > > > Hi Matthias,
> >
> > > > > Thanks for clarifying that. From the explication of the rules for
> > > > > confirms, it had appeared to me that the relationships to queues and
> > > > > consumers had more weight in the confirmation process. That not being
> > > > > the case, I have 2 brief follow up questions:
> >
> > > > > 1. Do consumer acks have more significance in transactions, such that
> > > > > I could use those to effect the functionality I'm looking for?
> >
> > > > > 2. If not, is there any way (apart from the admin HTTP API) for
> > > > > determining the number of consumers (e.g., bound queues) for a given
> > > > > exchange? This way I could test the number of reply messages as you
> > > > > suggest; otherwise, with needing to register and account for a dynamic
> > > > > array of servers, I would lose a good deal of what I had hoped would
> > > > > be the benefit of a fanout broadcast.
> >
> > > > > Thanks in advance,
> > > > > Jonathan
> >
> > > > > On Sep 23, 6:55 pm, Matthias Radestock <matth... at rabbitmq.com> wrote:
> > > > > > On 23/09/11 21:26, gneeri wrote:
> >
> > > > > > > My expectation, then, is that producer would publish its message and
> > > > > > > then be blocked until the delayed consumer returns an ack
> >
> > > > > > Confirms mean one thing and one thing only: that the broker has accepted
> > > > > > responsibility for the message.
> >
> > > > > > They concern the interaction between producers and the broker, and  have
> > > > > > nothing to do with consumer acks.
> >
> > > > > > Yes, there are some conditions in which a consumer ack can trigger a
> > > > > > confirm in RabbitMQ, but that is an implementation detail and not
> > > > > > something you'd want to design an application around.
> >
> > > > > > > My hope is that I will not have to aggregate a list of servers and
> > > > > > > pass along a single message to each server, but perhaps that is what
> > > > > > > will be required.
> >
> > > > > > You can still send a single message via a fanout exchange. But the
> > > > > > consumers then in turn need to send a message back to the sender. Your
> > > > > > sender will need to know how many consumers there are so it knows when
> > > > > > it has gotten all responses..
> >
> > > > > > Regards,
> >
> > > > > > Matthias.
> > > > > > _______________________________________________
> > > > > > rabbitmq-discuss mailing list
> > > > > > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > > _______________________________________________
> > > > > rabbitmq-discuss mailing list
> > > > > rabbitmq-disc... at lists.rabbitmq.com
> > > > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> > > > _______________________________________________
> > > > rabbitmq-discuss mailing list
> > > > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-disc... at lists.rabbitmq.com
> > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss