[rabbitmq-discuss] Bitcask Message Store Index

[Alvaro Videla]

Hi Matthew,

On Oct 13, 2011, at 12:32 AM, Matthew Sackman wrote:

> Huh, that's interesting. I'm curious though - how do the properties of
> your implementation differ from our default? - I'm not trying to
> discourage you at all, just curious. Even if it's just for gaining
> understanding and experience of implementing Rabbit's internal APIs,
> it's still valuable.

So my first goal was to learn more about the Rabbit's internal APIs. Besides that I haven't conducted any benchmarks to see what could be the advantage of using Bitcask over ETS.

According to the Basho devs they wanted to achieve this out of Bitcask:

- low latency per item read or written
- high throughput, especially when writing an incoming stream of random items
- ability to handle datasets much larger than RAM w/o degradation
- crash friendliness, both in terms of fast recovery and not losing data
- ease of backup and restore
- a relatively simple, understandable (and thus supportable) code structure and data format
- predictable behavior under heavy access load or large volume 
- a license that allowed for easy default use in Riak

From those items I think that the ability to handle datasets much larger than RAM w/o degradation could be an advantage over ETS. Still… If I understand correctly the datastore implementation I think when Rabbit start to get pressure on RAM usage it does some flushes to disk. Perhaps there could be a benefit there… not sure.

What I liked from this experiment is that there's no much code needed to have a different index implementation. Basho also created a LevelDB Erlang/C++ NIF, I think that can also be made a index quite easy. I've created a wrapper for it for the umbrella but didn't had time to also implement the index behaviour.

> Simplest way is to set the msg_store_index_module param in your
> rabbitmq.config file - check the rabbit.app to see how it's set, but
> there's nothing unusual there. Also check rabbitmq-toke which
> (re-)implements the same interface and probably changes the env slightly
> differently... ;)

Yes… in my plugin I'm using the rabbit_boot_step thing to set my plugin as the default index.

Cheers,

Alvaro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111013/fe14444d/attachment.htm>