[rabbitmq-discuss] Bitcask Message Store Index
matthias at rabbitmq.com
Thu Oct 13 09:46:06 BST 2011
On 13/10/11 08:57, Alvaro Videla wrote:
> So my first goal was to learn more about the Rabbit's internal APIs.
> [...] What I liked from this experiment is that there's no much code
> needed tohave a different index implementation.
This has been a great test of the msg_store_index API and its docs - you
managed to implement the bitcask index relatively straightforwardly. And
without needing to ask any questions ;)
> Besides that I haven't conducted any benchmarks to see what could be the
> advantage of using Bitcask over ETS.
The ets index is not a bottleneck performance wise. Its main shortcoming
is the per-message RAM cost.
> According to the Basho devs they wanted to achieve this out of Bitcask:
> - ability to handle datasets much larger than RAM w/o degradation
> From those items I think that the ability to handle datasets much
> larger than RAM w/o degradation could be an advantage over ETS.
For that to be effective the msg_store itself, not just the index, would
have to sit on top of bitcask.
bitcask has an in-memory index, very much like the
rabbit_msg_store_ets_index, so there would still be a per-message memory
cost. That cost might be a bit lower with bitcask, but not by more than
50%, since that's the total overhead.
I doubt bitcask would perform any better than the default msg_store,
since the latter has been highly tuned for queue-like access patterns.
One area where bitcask might excel is recovery time, particularly after
an unclean shutdown / crash.
More information about the rabbitmq-discuss