[rabbitmq-discuss] File changes from 1.6.0 to 1.7.0

[tsuraan]

> During the process of going through QA, we realised that mnesia is too
> flawed for us to use it as part of the persister. We were never going
> to store messages directly in mnesia, merely accountancy information.
> However, when operating in "disc_copies", mnesia keeps the entire table
> in RAM, and periodically writes the table out to disk. Unfortunately,
> it just writes the entire table out to disk, which means that for a
> large table with 99% static entries, but 1% rapidly changing entries,
> it will rewrite vast amounts of data unnecessarily, whenever it decides
> to write out the table. The other problem is that when running in
> "disc_only_copies", the entire table is disk only, using dets. Dets has
> an upper limit of 2GB of data, and performs very very badly - I've
> never seen it push a disk past about 10MB/s. These problems together
> have meant that in order to achieve the goals of zero per message RAM
> cost, and a sensible bound on the number of times data is rewritten, we
> have decided to move away from using mnesia for the persister.

I'm not sure if you saw the Dukes of Erl mnesiaex project
(http://code.google.com/p/mnesiaex).  I'm guessing you're well on your
way with a different strategy, but I've found tokyo to be really high
performance, and I'd be curious to hear how mnesia with a tokyo
backend works in a real project like rabbit.

> Unfortunately, these issues were only realised fairly late in the day,
> hence the delay whilst we do some redesigning and recoding.

That's always depressing.  Good luck getting this thing working; I'm
sure looking forward to it :)