[rabbitmq-discuss] About the new persistor

Wed Mar 10 10:37:59 GMT 2010

On Wed, Mar 10, 2010 at 06:22:52PM +0800, Alvaro Videla wrote:
> After checking the code of the new persister, I saw that It can be used as a K+V store, with memory cache and persistence to disk. Please correct me if I'm wrong.

You're not quite right there. The new persister is much more than just
the msg_store module. The c. 7000 new lines are in:
rabbit_msg_store.erl
rabbit_msg_file.erl
rabbit_msg_store_gc.erl
rabbit_queue_index.erl
rabbit_variable_queue.erl
bpqueue.erl
rabbit_msg_store_index.erl
rabbit_msg_store_ets_index.erl
rabbit_msg_store_misc.erl
file_handle_cache.erl
rabbit_memory_monitor.erl
rabbit_queue.hrl
rabbit_msg_store.hrl
rabbit_msg_store_index.hrl

Plus changes to rabbit_amqqueue_process.erl amongst others.

It really depends quite what you define as the "new persister". You're
not wrong though, in that the msg_store can be used as a K+V store (with
reference counting). However, it is very carefully designed to work
best with the access patterns that queues provide - it may not do so
well on randomised patterns.

> So, it would be nice if the module rabbit_msg_store can be used standalone. 
> In which other modules it depends?

xref is probably the best way to find out. But as a minimum:
rabbit_msg_store.erl
rabbit_msg_file.erl
rabbit_msg_store_gc.erl
rabbit_msg_store_index.erl
rabbit_msg_store_ets_index.erl
rabbit_msg_store_misc.erl
file_handle_cache.erl
rabbit_msg_store.hrl
rabbit_msg_store_index.hrl

And of course if you really want zero-RAM-cost-per-entry you should be
using the rabbitmq-toke plugin which pushes the index into Tokyo
Cabinet, which you'll also need.

> What do you guys think about it?

Not sure really. It hasn't really been designed with that in mind, so I
have no idea how well it'll work.

> I believe it could help easing the benchmarking of the persister too.

I don't really agree there - the queue_index also plays a crucial role
and writes to disk too. We've done and are doing extensive system-wide
testing, and as you've no doubt seen in rabbit_tests, there are some
large tests to stress the msg_store on its own.

Matthew