<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks Alex. That’s also something I thought of - creating a transaction log somewhere else and replay that log if rabbit storage gets corrupted.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">However wouldn’t I have the same the problem if I use a specific queue to push data to another datastore, i.e. if the consumer listening to that queue is not
fast enough, messages will accumulate in the queue. If queue data is corrupted shortly after, I will lose messages before my “transaction log writer” see them.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The only way that I can think of is to persist the message to my custom transaction log (MySQL, etc) _<i>on the publisher side</i>_ before message goes into
rabbit. In order to guarantee that _<i>all</i>_ messages go in the transaction log, I would have to do this synchronously, i.e. if I can’t persist the data in my transaction log, I won’t bother pushing that message to rabbit. But this seems to be doing too
much on the producer side and will obviously affect producer performance. Other way of achieving this would be to use two phase commits but I don’t want to go there…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I will definitely check @jbrisbin’s work. That sounds really interesting.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">oseymen</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Alex Lovell-Troy [mailto:alex.lovelltroy@gmail.com]
<br>
<b>Sent:</b> 03 June 2011 12:35<br>
<b>To:</b> Ozan Seymen<br>
<b>Cc:</b> rabbitmq-discuss@lists.rabbitmq.com<br>
<b>Subject:</b> Re: [rabbitmq-discuss] Database Corruption Possibilities<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There are a handful of approaches to this problem which all seem to follow a similar pattern. Until there is a way to replace mnesia with a replicable back end, we've all written something on our own that wraps each amqp transaction with
a transactional write to the datastore of our choice and. I've seen this done with MySQL, MongoDB, and riak personally. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">@jbrisbin has done some interesting work with implementing a specific queue that sends messages to riak that looks intriguing. I believe there is also a piece that pushes messages from riak into rabbitmq.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">At least, that was the state of the art the last time I looked<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Anyone else?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-alex<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On Fri, Jun 3, 2011 at 12:21 PM, Ozan Seymen <<a href="mailto:Ozan.Seymen@tdpg.com">Ozan.Seymen@tdpg.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi all,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Can someone please explain the scenarios where we might have Rabbit message storage (is it mnesia?) corrupted in a way that it is not recoverable?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">In the solution I am working on, I simply cannot afford to lose any messages. In order to secure this, I will:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p><span style="font-family:Symbol">·</span><span style="font-size:7.0pt">
</span>Rely on publisher confirms. This should ensure that broker will always confirm whether it assumed responsibility and persisted the message.
<o:p></o:p></p>
<p><span style="font-family:Symbol">·</span><span style="font-size:7.0pt">
</span>Implement durable queues/exchanges.<o:p></o:p></p>
<p><span style="font-family:Symbol">·</span><span style="font-size:7.0pt">
</span>Ack enabled in consumers to prevent losing messages if consumer dies halfway. I will solve the ordering problem on the consumer side.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Even though all of these above prevent message loss in normal conditions, none of them covers the case where data gets corrupted in the broker. There is a window (albeit small)
that things might go wrong: broker assumes responsibility (message is in the disk) and before message is sent to the consumer, broker experiences problems which corrupts the storage. If I can’t bring the broker up with all data previously persisted, I’ve lost
messages – producer forgot about the message as broker accepts responsibility and consumers have no idea about the message as it hasn’t been delivered to it.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Am I a total paranoid that is beyond help? Even so, I would really appreciate any info you guys can share.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Cheers,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">oseymen<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black">This e-mail and any attached files are intended for the named addressee only. It contains information which may be confidential and legally privileged and also protected
by copyright. Unless you are the named addressee (or authorised to receive for the addressee) you may not copy or use it or disclose it to anyone else. If you received it in error please notify the sender immediately and then delete it from your system.<br>
<br>
Please be advised that the views and opinions expressed in this e-mail may not reflect the views and opinions of The Digital Property Group Limited or any of its subsidiary companies.<br>
<br>
We make every effort to keep our network free from viruses. However, you do need to check this e-mail and any attachments to it for viruses as we can take no responsibility for any computer virus which may be transferred by way of this e-mail. We reserve the
right to monitor all e-mail communications.<br>
<br>
The Digital Property Group Limited is a Daily Mail and General Trust plc company.<br>
Registered Office: Northcliffe House, 2 Derry Street, London, W8 5TT.<br>
Registered in England & Wales No: 02290527 VAT no. 243571174<br>
</span><br>
______________________________________________________________________<br>
This email has been scanned by the MessageLabs Email Security System.<br>
For more information please visit <a href="http://www.messagelabs.com/email" target="_blank">
http://www.messagelabs.com/email</a> <br>
______________________________________________________________________<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><br>
______________________________________________________________________<br>
This email has been scanned by the MessageLabs Email Security System.<br>
For more information please visit <a href="http://www.messagelabs.com/email">http://www.messagelabs.com/email</a>
<br>
______________________________________________________________________<o:p></o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Black" size="1">This e-mail and any attached files are intended for the named addressee only. It contains information which may be confidential and legally privileged and also protected by copyright. Unless you are the named addressee
(or authorised to receive for the addressee) you may not copy or use it or disclose it to anyone else. If you received it in error please notify the sender immediately and then delete it from your system.<br>
<br>
Please be advised that the views and opinions expressed in this e-mail may not reflect the views and opinions of The Digital Property Group Limited or any of its subsidiary companies.<br>
<br>
We make every effort to keep our network free from viruses. However, you do need to check this e-mail and any attachments to it for viruses as we can take no responsibility for any computer virus which may be transferred by way of this e-mail. We reserve the
right to monitor all e-mail communications.<br>
<br>
The Digital Property Group Limited is a Daily Mail and General Trust plc company.<br>
Registered Office: Northcliffe House, 2 Derry Street, London, W8 5TT.<br>
Registered in England & Wales No: 02290527 VAT no. 243571174<br>
</font>
<BR>
______________________________________________________________________<BR>
This email has been scanned by the MessageLabs Email Security System.<BR>
For more information please visit http://www.messagelabs.com/email <BR>
______________________________________________________________________<BR>
</body>
</html>