[rabbitmq-discuss] Custom storage backends for queues (MySQL)

Tue Jan 18 20:58:54 GMT 2011

Hi, Jon...

> A couple quick questions about this:
>
> Should  a replacement  rabbit_backing_queue be  packaged as  a regular
> plugin?

I don't see why not; this is how I've packaged the skeleton of what
I'm doing with MySQL.  John D plans to do the same but for the moment
is working in place in his server directory since it was convenient to
get started prototyping and exploring that way.  I imagine he'll
consume my packaging into his working branch shortly.

> How would I configure the broker to use my own storage module?

You need to edit your rabbitmq.config file a la:

[
  {rabbit, [{backing_queue_module, rabbit_weirdball_queue}]}
].

...and of course make sure your plugin is appropriately packaged with
your backing queue module and its dependencies in it, and then either
present in, or symlinked from, your broker's plugins directory.  That 
part's pretty easy, modulo potentially some minor Erlang load path 
gremlins.

> Should one start with rabbit_variable_queue.erl and replace only the
> bits that talk to the message store (leaving the memory management
> logic intact) as a first step?

The easy answer there is a bit less obvious, and may depend on what
your objectives are for your storage plugin.  If others watching have
opinions or thoughts, they should jump in.

The rabbit_variable_queue implementation is fairly complex, and
contains a lot of logic for things like monitoring message rates and
tweaking writes to the rabbit_msg_store in response to them.  Your
plugin may or may not want to do that in exactly the same way (or at
all).  One reason, for example, that one might want to write a storage
plugin would be to inherit some reliability or recovery properties
from whatever storage system lies behind it, in which case you might
want to write more things more often than r_v_q does.

John's been hacking on an Mnesia-based backing queue which he started
from a very stripped down r_v_q.  There's also a sort of base case-ish 
"RAM queue" extractable from that which shows a nice path to getting a
working minimal backing_queue implementation that can keep a broker 
running and passing tests etc. while one works from it.  We can chat 
about such things when we talk on the phone.  It might be instructive 
to look at his stuff once he's done a bit more tidying on it.

> It would probably be a good idea to look at the content type of the
> message to figure out how to store it in the backing store, right
> (which would open the possibility of working with message data
> independently of the broker by interrogating its contents)?

Very possibly.  In fact mapping the contents of a stored message to a
representation suitable for a given back end seems like it would
require varying amounts and types of thought tailored to the storage
system in question to minimize awkward impedance mismatches,
performance problems, correctness issues, etc.

> Would it be a bad idea to have store-side code as part of the storage
> system? In particular, I'm thinking about having Erlang-language M/R
> code or event triggers which would need to be installed into the
> server (Riak in this case) to manage indexes and metadata. In general,
> I don't like the idea of having to install things in more than one
> place. It would probably make some things considerably easier (and
> more performant), though.

I'm definitely not sure but very interested in talking further and
learning more about what you have in mind?

I'll dredge up our phone conference info, and get back to you 
separately...

Best regards,
Jerry