[rabbitmq-discuss] RabbitMQ 1.7 and persistence limit?

Tue Oct 13 13:34:07 BST 2009

Lynton,

On Tue, Oct 13, 2009 at 02:15:46PM +0200, Lynton Grice wrote:
> The SYNC transactions are not my concern, I do not really mind if those
> messages get lost, it is the ASYNC ones that I want 100% guarantee they will
> never get lost.

I'm completely confused by what you mean by this. That is just not
possible. If you send an async message, eg basic.publish, there are
probably a good few dozen buffers in your local computer, the language
you're writing in, the OS network stack, the network card itself, and so
forth, all in which the message can be stored and lost in the event of
power failure. Not to mention buffers in network hubs/switches/routers,
and then in Rabbit itself, all of which can hold the message
temporarily, after it's been "sent" and before it's been written to
disk.

Unless you get some sort of reply from Rabbit, which says that "yes,
I've received the message and it's been written to disk", there is *no*
guarantee that any message you send has even been received by Rabbit,
let alone safely stored.

> You said " With the persister in 1.7, messages are kept in RAM but are also
> written out to disk." - are the messages written to disk by default as well?
> As in the choice between "ram_copies", "disk_copies" and "disc_only_copies"?
> Or are the messages simple written anyway in the 1.7 persister?

By definition, persistent messages are written to disk. They also happen
to be stored in RAM in the persister in 1.7 (and before), for a variety
of reasons. This has *nothing* to do with the various modes of operation
of mnesia, as the persister does not use mnesia at all.

> Using transactions sounds like the answer for me for now, thanks.

Right, and thus this is effectively synchronous messaging as you get the
reply back from Rabbit - the commit-ok - when you commit on the channel.

> Last question: Do you ever forsee a day when there is no 2GB limit to DETS?

It's possible that the Erlang/OTP folks will get round to rewriting it
to make it 64-bit safe, I don't know. Certainly it's not something we're
looking at doing.

Matthew