[rabbitmq-discuss] File changes from 1.6.0 to 1.7.0

[Matthew Sackman]

On Tue, Sep 29, 2009 at 09:47:55AM -0500, tsuraan wrote:
> I guess that still answers my question well enough; I was assuming the
> new persister was part of 1.7, so whenever the new persister comes
> out, I'll have to do that.  Why was it pushed back?  I thought it had
> passed QA a few months ago.

During the process of going through QA, we realised that mnesia is too
flawed for us to use it as part of the persister. We were never going
to store messages directly in mnesia, merely accountancy information.
However, when operating in "disc_copies", mnesia keeps the entire table
in RAM, and periodically writes the table out to disk. Unfortunately,
it just writes the entire table out to disk, which means that for a
large table with 99% static entries, but 1% rapidly changing entries,
it will rewrite vast amounts of data unnecessarily, whenever it decides
to write out the table. The other problem is that when running in
"disc_only_copies", the entire table is disk only, using dets. Dets has
an upper limit of 2GB of data, and performs very very badly - I've
never seen it push a disk past about 10MB/s. These problems together
have meant that in order to achieve the goals of zero per message RAM
cost, and a sensible bound on the number of times data is rewritten, we
have decided to move away from using mnesia for the persister.

Unfortunately, these issues were only realised fairly late in the day,
hence the delay whilst we do some redesigning and recoding.

Matthew