[rabbitmq-discuss] Database Corruption Possibilities

[Alex Lovell-Troy]

There are a handful of approaches to this problem which all seem to follow a
similar pattern.  Until there is a way to replace mnesia with a replicable
back end, we've all written something on our own that wraps each amqp
transaction with a transactional write to the datastore of our choice and.
 I've seen this done with MySQL, MongoDB, and riak personally.

@jbrisbin has done some interesting work with implementing a specific queue
that sends messages to riak that looks intriguing.  I believe there is also
a piece that pushes messages from riak into rabbitmq.

At least, that was the state of the art the last time I looked

Anyone else?

-alex

On Fri, Jun 3, 2011 at 12:21 PM, Ozan Seymen <Ozan.Seymen at tdpg.com> wrote:

>  Hi all,
>
>
>
> Can someone please explain the scenarios where we might have Rabbit message
> storage (is it mnesia?) corrupted in a way that it is not recoverable?
>
>
>
> In the solution I am working on, I simply cannot afford to lose any
> messages. In order to secure this, I will:
>
>
>
> ·         Rely on publisher confirms. This should ensure that broker will
> always confirm whether it assumed responsibility and persisted the message.
>
> ·         Implement durable queues/exchanges.
>
> ·         Ack enabled in consumers to prevent losing messages if consumer
> dies halfway. I will solve the ordering problem on the consumer side.
>
>
>
> Even though all of these above prevent message loss in normal conditions,
> none of them covers the case where data gets corrupted in the broker. There
> is a window (albeit small) that things might go wrong: broker assumes
> responsibility (message is in the disk) and before message is sent to the
> consumer, broker experiences problems which corrupts the storage. If I can’t
> bring the broker up with all data previously persisted, I’ve lost messages –
> producer forgot about the message as broker accepts responsibility and
> consumers have no idea about the message as it hasn’t been delivered to it.
>
>
>
> Am I a total paranoid that is beyond help? Even so, I would really
> appreciate any info you guys can share.
>
>
>
> Cheers,
>
> oseymen
>
>
>
>
>
>
>
>
>
> ------------------------------
> This e-mail and any attached files are intended for the named addressee
> only. It contains information which may be confidential and legally
> privileged and also protected by copyright. Unless you are the named
> addressee (or authorised to receive for the addressee) you may not copy or
> use it or disclose it to anyone else. If you received it in error please
> notify the sender immediately and then delete it from your system.
>
> Please be advised that the views and opinions expressed in this e-mail may
> not reflect the views and opinions of The Digital Property Group Limited or
> any of its subsidiary companies.
>
> We make every effort to keep our network free from viruses. However, you do
> need to check this e-mail and any attachments to it for viruses as we can
> take no responsibility for any computer virus which may be transferred by
> way of this e-mail. We reserve the right to monitor all e-mail
> communications.
>
> The Digital Property Group Limited is a Daily Mail and General Trust plc
> company.
> Registered Office: Northcliffe House, 2 Derry Street, London, W8 5TT.
> Registered in England & Wales No: 02290527 VAT no. 243571174
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110603/51fa024b/attachment-0001.htm>