[rabbitmq-discuss] New Persister merged into default

[Matthew Sackman]

On Thu, Aug 12, 2010 at 08:31:20AM -0400, Aaron Westendorf wrote:
> Bummer, that's a big surprise.  I'd be curious to see a comparison
> with a persister backed by a storage engine that does use mmap, like
> Tokyo Cabinet.  Not that I doubt the team's abilities, we just have
> had fantastic results with applications such as Varnish, TC and Mongo
> which all make extensive use of mmap and let the kernel do all the
> heavy lifting.

Bear in mind that we don't have many random seeks. All our writes are
append only, and queues only have one read head. This doesn't make life
easy, but it does mean that predicting where the next read is going to
be is rather easy. Plus obviously using block reads etc etc... Given the
way queues are accessed, I'm not convinced you'd see enormous speedups
using mmap. Oh and also, you tend to have many writes for a message, and
0 or 1 reads for a message. So all those stores that are optimised for
heavy read loads - well, needless to say they don't tend to maintain
their performance when presented with a high write-to-read ratio.

> Maybe with the abstraction layer in place, we'll take
> that on the next time we have a hack-a-thon day.

Please do - the Toke Erlang driver for Tokyo Cabinet that I wrote should
serve you well there.

> I'd also be curious
> to try an extension of the persistent messaging, so that after X
> seconds it can be thrown out.

If you get anywhere with that, please let us know! We have plans to do a
per-queue TTL (no message in the queue can have been in the queue for
more than N seconds), but general per message TTL is very very much
harder, which is why we've not done it.

Matthew