[rabbitmq-discuss] Performance of "blocking publisher-confirms" vs. transactions

[Matt Pietrek]

Hi Simone,

| Can you describe why you think publisher confirms are not appropriate in
your scenario?

I think if you read the paragraph that starts with "At the heart of things"
it describes the scenario in enough detail. In a follow up, Simon MacMullen
appears to agree that publisher confirms aren't ideal for us.

In short, when we publish a message, we must know that the broker safely
has it. With transactions, OR a blocking publisher-confirm model, the
client is assured it has it.

HOWEVER, all the publisher-confirm descriptions/examples I've seen assume
an async model: Publish a message and some time later you'll be told that
the message is safely in the broker's hands. However, that window between
those two events is the crux of the problem in our environment.

Assume we went with the async publisher-confirm model as suggested. What if
our client dies and is not around to re-send the message? Or what if the
client dies, and milliseconds later the broker dies? No publisher confirm
would be forthcoming, and hence, lost messages.

Top level problem for us: Our state machine, which persists every critical
change of state to a database, will be incorrect. We will have recorded
"Yup, that message was sent", when in reality it might not have made it to
broker, or the broker died before the confirm could be sent.

By not recording that we've sent the message until AFTER a
transaction/publisher-confirm completes, we can guarantee the integrity of
our state-machine. However, as we've all seen, transactions impose a limit
on message throughput. I'm simply trying to understand if there's any
non-trivial message rate difference expected between using a transaction
vs. waiting in my publish routine for the confirm.


| If you happen to be using the .NET API this is already implemented, the
WaitForConfirms and WaitForConfirmsOrDie on the IModel interface provide
this behavior.

Unfortunately, I'm not on a .NET client. I'm using Python/Pika. I'd
consider going to the c-library (wrapped from Python) if there's some
non-trivial benefit to using a synchronous publisher-confirm.

Thanks,

Matt

On Thu, Feb 16, 2012 at 1:08 PM, Simone Busoli <simone.busoli at gmail.com>wrote:

> Hi Matt, I've gone through the old thread very quickly so forgive me if
> I'm missing something. Your first sentence leaves me with some doubts:
>
> As I've detailed in other posts to this DL (http://bit.ly/zf6sKC), we
>> have situation where straightforward publisher-confirms aren't appropriate
>> in our scenario. We need to know the broker has received the message, and
>> we have to assume our client can die at any moment, so retransmitting
>> messages is not an option for us.
>>
>
> Publisher confirms guarantee that the broker has taken care of the message
> and it has been either delivered to a queue or to a consumer. This is a
> strong delivery guarantee, even if the client dies. Coupled with persistent
> messages and durable exchange/queues, it guarantees at-least-once delivery.
> Can you describe why you think publisher confirms are not appropriate in
> your scenario?
>
>
>> What I'd like to know is: Is there any benefit to creating our own
>> "blocking publisher-confirm publishing"? That is, we'd do the following:
>>
>> Publish a single message
>> Block on an event 'X'
>> When the publisher confirm comes in, signal 'X'
>> Return from the publish call
>>
>
> If you happen to be using the .NET API this is already implemented, the
> WaitForConfirms and WaitForConfirmsOrDie on the IModel interface provide
> this behavior.
>
>
>> In a mirrored queue scenario, is there any advantage to doing this vs.
>> using straight-up transactions?
>>
>> Matt
>>
>> p.s. I realize in the above pseudocode that multiple threads would be
>> necessary - One to block and the other to listen for the publisher-confirm.
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120216/9d241271/attachment.htm>