[rabbitmq-discuss] Pub/Sub -- block publisher until all acks received from subscribers?

[Nick Martin]

Thanks, Emile, for your explanation. That was helpful.

An alternative approach would be to declare a return queue for the
> consumers to report their progress to the publisher. The publisher can
> subscribe to this queue and use this to wait until the subscribers are
> ready before proceeding. The publisher would need to know how many
> subscribers to expect replies from.
>

This is one of the main reasons I wanted to use pub/sub. I don't want the
publisher to have to know how many subscribers exist.


> If your design allows it you should avoid coupling producers and consumers
> as tightly as you seem to need here. What is the reason for forcing
> producers and consumers in lockstep?


Here's a quick explanation of my design. I have a system with multiple
machines, each with several worker threads. Each worker takes tasks from a
shared work queue one at a time, does that task and returns for another
task. That part of the system already works well. The new part I am trying
to create is a control system. The control system will send pause and
resume messages to a worker manager that runs on each worker machine. The
worker manager is the subscriber. I would like the worker manager to wait
until all of its workers are paused before acknowledging the pause message
to the publisher. Sometimes pausing a worker can take up to a minute.

Since each worker manager will wait to send its ack to the publisher until
all of its workers are paused, the publisher will know that all of the
workers throughout the system are paused and can safely continue. Later the
publisher will send a resume message to all of the worker managers and work
can resume.

Does this seem like a reasonable design? In the future it would probably be
nice to have a system that has more holistic view of its workers, but for
now I am not concerned if the publisher sends messages to one machine or
100.


> Can the same aim be achieved in another way by taking a larger view of the
> system?


I'm not sure. Do you have any suggestions?

Thanks,
Nick


On Thu, Nov 1, 2012 at 3:42 AM, Emile Joubert <emile at rabbitmq.com> wrote:

>
> Hi Nick,
>
> This page has more detail and you may find some of your answers:
> http://www.rabbitmq.com/**confirms.html<http://www.rabbitmq.com/confirms.html>
>
>
> On 31/10/12 22:10, Nick Martin wrote:
>
>> Wait until all messages published since the last call have been
>> either ack'd or nack'd by the broker.
>>
>
> A message being ack'd by the broker is different from being ack'd by a
> consumer. If the broker acknowledges a messages it means that the broker
> has taken responsibility for it by writing it to disk or successfully
> handed it off. Handing off could mean returning the messages to the
> publisher (unroutable mandatory msg) or successfully delivering to
> consumer. The publisher cannot choose how the broker discharges this
> responsibility. In your case you would like a broker acknowledgement only
> when the message is successfully delivered to a consumer, but the broker
> may decide to write the message to disk and acknowledge when the OS
> guarantees that the disk write succeeded.
>
>
>  It seems to me that the publisher is
>> supposed to block until all of the messages are ack'd via its call to
>> waitForConfirmsOrDie(). Is this what this is really supposed to do?
>>
>
> Yes, waitForConfirmsOrDie() will block until an ack is received from the
> broker.
>
>
>  This example code seemed like it matched up perfectly with what I was
>> trying to do. But, it doesn't seem to work the way I thought it did. In
>> fact, if, in the consumer thread, I turn off auto-acking messages, then
>> waitForConfirmsOrDie() still returns immediately.
>>
>
> This is because the broker has taken responsibility for the message by
> writing it to disk.
>
>
>  So what does waitForConfirmsOrDie() actually do? When would it block?
>>
>
> It could block e.g. as a result of attempting to perform an fsync
> operation.
>
>
>  If waitForConfirmsOrDie doesn't do what I want, is there a way to make a
>> publisher wait until all subscribers ack a message before proceeding?
>>
>
> An alternative approach would be to declare a return queue for the
> consumers to report their progress to the publisher. The publisher can
> subscribe to this queue and use this to wait until the subscribers are
> ready before proceeding. The publisher would need to know how many
> subscribers to expect replies from.
>
> If your design allows it you should avoid coupling producers and consumers
> as tightly as you seem to need here. What is the reason for forcing
> producers and consumers in lockstep? Can the same aim be achieved in
> another way by taking a larger view of the system?
>
>
>
>
> -Emile
>
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121101/49cf64b0/attachment.htm>