[rabbitmq-discuss] Bitcask Message Store Index

Thu Oct 13 09:46:06 BST 2011

Alvaro,

On 13/10/11 08:57, Alvaro Videla wrote:
> So my first goal was to learn more about the Rabbit's internal APIs.
>  [...] What I liked from this experiment is that there's no much code
> needed tohave a different index implementation.

This has been a great test of the msg_store_index API and its docs - you 
managed to implement the bitcask index relatively straightforwardly. And 
without needing to ask any questions ;)

> Besides that I haven't conducted any benchmarks to see what could be the
> advantage of using Bitcask over ETS.

The ets index is not a bottleneck performance wise. Its main shortcoming 
is the per-message RAM cost.

> According to the Basho devs they wanted to achieve this out of Bitcask:
>  [...]
> - ability to handle datasets much larger than RAM w/o degradation
> [...]
>  From those items I think that the ability to handle datasets much
> larger than RAM w/o degradation could be an advantage over ETS.

For that to be effective the msg_store itself, not just the index, would 
have to sit on top of bitcask.

bitcask has an in-memory index, very much like the 
rabbit_msg_store_ets_index, so there would still be a per-message memory 
cost. That cost might be a bit lower with bitcask, but not by more than 
50%, since that's the total overhead.

I doubt bitcask would perform any better than the default msg_store, 
since the latter has been highly tuned for queue-like access patterns. 
One area where bitcask might excel is recovery time, particularly after 
an unclean shutdown / crash.

Regards,

Matthias.