<font face="trebuchet ms,sans-serif">Hello Matt, thanks for the explanation and sorry for not reading the previous thread carefully, I did now. </font><div><font face="'trebuchet ms', sans-serif"><br></font><div><font face="trebuchet ms,sans-serif">I can't tell you what differences you should expect between using a blocking publisher confirms model and transactions as I've never used transactions before, but allow me to insist a bit more on why the first may fit your scenario by describing mine. I'm using RabbitMQ in an environment where high reliability, together with strict ordering of messages and exactly-once quality of service are needed. We're using asynchronous publisher confirms to achieve that.</font></div>
<div><font face="trebuchet ms,sans-serif"><br></font></div><div><font face="trebuchet ms,sans-serif">You seem to be worried about a scenario in which you believe some messages to be in the hands of the server while instead they aren't due to the asynchronous nature of publisher confirms, and especially that the publishers may not be there to republish them in case something goes wrong. </font></div>
<div><font face="trebuchet ms,sans-serif">Without knowing much about your system I would suggest that you don't consider the messages as "delivered" until you receive the asynchronous confirm from the broker, and just pretend they were never published until this happens. The problem you describe seems to arise because you consider published messages as "delivered", which is not true until you receive a confirm.</font></div>
<div><font face="trebuchet ms,sans-serif">So what's worst case scenario difference between publisher confirms used in this way compared to transactions? If a publisher dies, in the first case you would consider as published only the messages that have been confirmed, thus pretending the unconfirmed ones to have never existed, in the second case an hypothetical ongoing transaction is rolled back. I think the difference here lies only in the number of messages you have tried to publish which may not have reached the broker. In the case of transactions it is at most 1, with async publisher confirms it may be more than 1 (it will be 1 with synchronous publisher confirms), but I don't think this number changes the nature of the problem.</font></div>
<div><br><div class="gmail_quote">On Thu, Feb 16, 2012 at 22:44, Matt Pietrek <span dir="ltr"><<a href="mailto:mpietrek@skytap.com">mpietrek@skytap.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Simone,<br><br><font face="'trebuchet ms', sans-serif"><div class="im">| Can you describe why you think publisher confirms are not appropriate in your scenario?<br><br></div>I think if you read the paragraph that starts with "At the heart of things" it describes the scenario in enough detail. In a follow up, Simon MacMullen appears to agree that publisher confirms aren't ideal for us.<br>
<br>In short, when we publish a message, we must know that the broker safely has it. With transactions, OR a blocking publisher-confirm model, the client is assured it has it.<br><br>HOWEVER, all the publisher-confirm descriptions/examples I've seen assume an async model: Publish a message and some time later you'll be told that the message is safely in the broker's hands. However, that window between those two events is the crux of the problem in our environment.<br>
<br>Assume we went with the async publisher-confirm model as suggested. What if our client dies and is not around to re-send the message? Or what if the client dies, and milliseconds later the broker dies? No publisher confirm would be forthcoming, and hence, lost messages.<br>
<br>Top level problem for us: Our state machine, which persists every critical change of state to a database, will be incorrect. We will have recorded "Yup, that message was sent", when in reality it might not have made it to broker, or the broker died before the confirm could be sent.<br>
</font><font face="'trebuchet ms', sans-serif"><br>By not recording that we've sent the message until AFTER a transaction/publisher-confirm completes, we can guarantee the integrity of our state-machine. However, as we've all seen, transactions impose a limit on message throughput. </font><font face="'trebuchet ms', sans-serif"> I'm simply trying to understand
if there's any non-trivial message rate difference expected between
using a transaction vs. waiting in my publish routine for the confirm.</font><br><font face="'trebuchet ms', sans-serif"><div class="im"><br><br>| If you happen to be using the
.NET API this is already implemented, the WaitForConfirms and
WaitForConfirmsOrDie on the IModel interface provide this behavior.<br><br></div>Unfortunately, I'm not on a .NET client. I'm using Python/Pika. I'd consider going to the c-library (wrapped from Python) if there's some non-trivial benefit to using a synchronous publisher-confirm.<br>
<br>Thanks,<br><br>Matt<br></font><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote">On Thu, Feb 16, 2012 at 1:08 PM, Simone Busoli <span dir="ltr"><<a href="mailto:simone.busoli@gmail.com" target="_blank">simone.busoli@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<font face="trebuchet ms,sans-serif">Hi Matt, I've gone through the old thread very quickly so forgive me if I'm missing something. Your first sentence leaves me with some doubts:<br></font><br><div class="gmail_quote">
<div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word"><div>As I've detailed in other posts to this DL (<a href="http://bit.ly/zf6sKC" target="_blank">http://bit.ly/zf6sKC</a>), we have situation where straightforward publisher-confirms aren't appropriate in our scenario. We need to know the broker has received the message, and we have to assume our client can die at any moment, so retransmitting messages is not an option for us.</div>
</div></blockquote><div><br></div></div><div><font face="'trebuchet ms', sans-serif">Publisher confirms guarantee that the broker has taken care of the message and it has been either delivered to a queue or to a consumer. This is a strong delivery guarantee, even if the client dies. Coupled with persistent messages and durable exchange/queues, it guarantees at-least-once delivery. Can you describe why you think publisher confirms are not appropriate in your scenario?</font></div>
<div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word"><div></div><div>What I'd like to know is: Is there any benefit to creating our own "blocking publisher-confirm publishing"? That is, we'd do the following:</div>
<div><br></div><div>Publish a single message</div><div>Block on an event 'X'</div><div>When the publisher confirm comes in, signal 'X'</div><div>Return from the publish call</div></div></blockquote><div><br>
</div></div><div><font face="'trebuchet ms', sans-serif">If you happen to be using the .NET API this is already implemented, the WaitForConfirms and WaitForConfirmsOrDie on the IModel interface provide this behavior.</font></div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word"><div>In a mirrored queue scenario, is there any advantage to doing this vs. using straight-up transactions?</div>
<span><font color="#888888"><div><br></div><div>Matt</div></font></span><div><br></div><div>p.s. I realize in the above pseudocode that multiple threads would be necessary - One to block and the other to listen for the publisher-confirm.</div>
<div><br></div></div>
<br></div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br>
</blockquote></div><br>
</div></div></blockquote></div><br></div></div>