[rabbitmq-discuss] acks from temporary queues

[gneeri]

Hi Alex,

So to follow up, I have run some tests with durable queues and
waitForConfirms, which are not giving me the results I would expect.

The test is set up as follows:

1. I have a producer publishing with confirmSelect and waitForConfirms
to a fanout exchange

2. I have 2 consumers, attached to durable queues bound to the fanout
exchange

3. both consumers have autoAck set to false and use manual basicAck
after processing

4. I have put a delay (via Thread.sleep()) into one of the consumers

My expectation, then, is that producer would publish its message and
then be blocked until the delayed consumer returns an ack to queue
which would in turn send an ack to the fanout exchange, followed by an
ack to the producer. This, however, does not appear to be the case and
the producer appears to be executing code below the waitForConfirms()
call, well before the delayed consumer sends back its ack. For good
measure, I have also tried placing the sleep() call before the
consumer calls nextDelivery() in case the simple acceptance of the
payload from the queue would be enough to pass acks back up the chain
to the producer. This too, did not appear to work.

Ultimately, then, my sense is that I have misconstrued your earlier
remarks. Earlier (below), I passed along the intended functionality.
My hope is that I will not have to aggregate a list of servers and
pass along a single message to each server, but perhaps that is what
will be required. In any case, if you could let me know what I am
missing in this test (i.e., if there is a correctable way of making
this work) or if I need to attempt an implementation based on one
message per server, I would very much appreciate it.

Thanks in advance,
Jonathan

On Sep 23, 12:19 pm, gneeri <gne... at gmail.com> wrote:
> Hi Alex,
>
> Thank you very much for your detailed reply. It is quite helpful and I
> do have one specific question with respect to your notes, but think it
> might be most useful if I briefly respond to your question about the
> intended system before inquiring further:
>
> The system is composed of a dynamic array of servers, each of which
> support user logins. In order to prevent data corruption (i.e.,
> concurrency problems), my intent is that, upon login to any of the
> servers, the server logged into broadcasts a message out to the other
> servers, telling them to revoke control over the logging in user's
> data. Then, and only once all of the other servers (subscribed to the
> login exchange at startup) have ack'd the request to relinquish
> control, do I want the login server to proceed and take control of the
> user's data.
>
> From there, I expect the blocking requirement is pretty clear. Also,
> insofar as the array of servers is dynamic, I'm hoping my desire to
> use a single message over a "fanout" exchange will make some sense (as
> opposed to having to send a single message to each server via a named
> queue (or perhaps routing key) that would require me to have a list of
> all of the servers in advance of publishing).
>
> Which brings me to my question(s): when you say: "The broker, in turn,
> will only acknowledge the message if . . . 2) it has been delivered to
> a consumer on each of the queues it had reached"--does this mean that
> the broker (in this case, fanout exchange) will only send an ack to
> the publisher after a consumer on each of the attached queues has been
> reached? After some very rough testing with multiple consumers
> attached to the same fanout exchange, it has appeared to me that the
> publisher may get an ack prior to one of the consumers sending its ack
> back to the broker. This may just be the result of bad test code (on
> my part) *or* it may relate the fact that I am using temporary queues
> (e.g., channel.queueDeclare().getQueue()) rather than durable queues,
> as you suggest below. If that latter is the case, are you suggesting
> that using durable queues would make the broker wait on all consumers
> for all of the queues before returning an ack to the publisher? If so,
> that sounds like it would do the trick.
>
> Sorry again, for being so verbose and thanks very much for your help.
>
> Cheers,
> Jonathan
>
> On Sep 23, 5:58 am, Alexandru Scvorţov <alexan... at rabbitmq.com> wrote:
>
>
>
>
>
>
>
> > Hi Jonathan,
>
> > > I apologize if this is an ill-informed question: In a nutshell, the
> > > thing I am trying to accomplish is blocking a producer until it has
> > > received acks (or even nacks) from all consumers on temporary queues
> > > bound to a given (fanout) exchange for a single message. From the
> > > docs, my sense is that using confirms and waitForConfirms would be
> > > preferable to full on transactions,
>
> > Confirms are usually the way to go.
>
> > With confirms enabled, calling waitForConfirms would block the producer
> > until all published messages on that channel are acknowledged *by the
> > broker*.  The broker, in turn, will only acknowledge the message if 1)
> > it has been persisted to disk; *or* if 2) it has been delivered to a
> > consumer on each of the queues it had reached.
>
> > You can't really specify that you only want the acknowledgement if the
> > messages are delivered and not if they're persisted.
>
> > Also, you said "temporary queues".  Confirms will only work as described
> > above if the message is persistent and if the queues it lands on are
> > durable.  A message that lands on a non-durable queue is considered
> > acknowledged on that queue immediately (of course, the broker will still
> > wait for it to be persisted or acknowledged on all the other queues it
> > reached, before acknowledging it to the producer).
>
> > > but as each instance of the
> > > producer will be on its own thread,
>
> > That's fine.  Just make sure to use a separate channel for each
> > producer (calling waitForConfirms on the same channel from multiple
> > threads will almost certainly not do what you want it to).
>
> > > The main point of confusion for me here is whether I can get multiple
> > > acks (from all consumers) for one message without having any prior
> > > information about the number of queues attached to the exchange. That
> > > is, I don't just want to know that one queue has received the message,
> > > but that all queues bound to the exchange have done so. It seems clear
>
> > Not quite sure what you mean.  A published message, even if it gets sent
> > to multiple queues, is still *one* message with a certain sequence
> > number.  A published message will be acknowledged by the broker only
> > once, when it has been written to disk or when it has been successfully
> > sent to a consumer from each queue.
>
> > Consumer acks and publisher acks are completely different things.  A
> > consumer acks a message to tell *the broker* (not the publisher) that it
> > has taken responsibility for the message (by using it up, by forwarding
> > it, by doing something with it).  The broker acks a message to tell *the
> > publisher* that it has taken responsibility for the message (by saving
> > it to disk, or by handing it over to consumers).
>
> > When a message reaches a queue, and is then delivered to a consumer, the
> > broker will consider the message delivered only when the consumer
> > acknowledges it.  So, if it weren't for the "writing the message to
> > disk" part, it would work exactly the way you're expecting it to.
>
> > What sort of system are you designing?
>
> > Hope this helps.
>
> > Cheers,
> > Alex
>
> > On Thu, Sep 22, 2011 at 12:22:50PM -0700, gneeri wrote:
> > > I apologize if this is an ill-informed question: In a nutshell, the
> > > thing I am trying to accomplish is blocking a producer until it has
> > > received acks (or even nacks) from all consumers on temporary queues
> > > bound to a given (fanout) exchange for a single message. From the
> > > docs, my sense is that using confirms and waitForConfirms would be
> > > preferable to full on transactions, but as each instance of the
> > > producer will be on its own thread, I am open to using transactions.
> > > The main point of confusion for me here is whether I can get multiple
> > > acks (from all consumers) for one message without having any prior
> > > information about the number of queues attached to the exchange. That
> > > is, I don't just want to know that one queue has received the message,
> > > but that all queues bound to the exchange have done so. It seems clear
> > > that if I send one message per attached queue, I could create a set of
> > > seqNos and check against that to verify acks for all of the messages,
> > > but without a list of bound queues (which I cannot see how to get),
> > > that does not seem like an option. In any case, it seems likely that
> > > there is something that I am missing here. That said, I would
> > > appreciate any thoughts (or even clarifying questions) on this.
>
> > > Thanks,
> > > Jonathan
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-disc... at lists.rabbitmq.com
> > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss