[rabbitmq-discuss] RabbitMQ 1.7 and persistence limit?

[Matthew Sackman]

On Tue, Oct 13, 2009 at 12:56:56PM +0200, Lynton Grice wrote:
> A long time ago I heard Mnesia has a 2GB storage limit.

That information is not correct. Mnesia has several different ways of
operating a table:

1. ram_copies. The table is stored in an ets table in memory and is
never written to disk. The only limit is the amount of available memory.
No 2GB limit here.

2. disc_copies. The table is stored in an ets table in memory, and is
periodically written to disk using the disk_log module (glad to see even
within Erlang's only libraries, they can't decide how to spell disk).
Again, there is no limit in size.

3. disc_only_copies. The table is stored in a dets table only on disk.
Dets tables have 32-bit internal pointers within the file, so you are
limited to 2GB.

Wonderfully, this means that if you happen to have a table in
disc_copies mode which is > 2GB, and you then decide you want to save
memory and force it to disc_only_copies, you blow up the table as dets
can't cope with tables that big.

> My question is what
> happens in the case were a company wanting to use RabbitMQ needs larger
> persistence? Say like 8GB?

The persister does not use Mnesia in any way. This is true of the
current persister in 1.7 and the future persister. We are not using
mnesia for the persister entirely because of its limitations.

The persister in 1.7 keeps all messages in memory, so you're entirely
bound by the amount of RAM available. The new persister will not have
this limitation and will be able to scale much better than the current
persister. I realise that we've been making noises about the "new"
persister since April, and we had hoped to get it out with 1.7, but hey,
software development is never a predictable activity!

> I have just installed the 1.7 release and it is super, great work guys!!

Good to hear! :D

Matthew