[rabbitmq-discuss] Last value cache for pub/sub pattern

[Ben Hood]

Bill,

On Fri, Nov 7, 2008 at 4:08 PM, Bill Soudan <bill at soudan.net> wrote:
> I see in the FAQ that support for a last value cache is planned:

Yes, well, many things are *planned* :-)

> I'd like to use AMQP and I need this feature, so I'm going to take a crack at implementing it.

Although this kind of thing is not (yet?) specified in the protocol, I
don't think it would too hard to do as a server specific extension.

> I do have one question though:  isn't there a race with the described implementation?  It seems possible to me that a new message may arrive from the bound exchange before the 'last-value' cache service responds with the most recent value.  Yes, I could fix this with some some of timestamp value or sequence value, but I would rather not push that responsibility onto the client.

I don't think that it would be an advantage to do this in the client,
since you can do this already. Also, the reason behind last value
caching is to be able to purge (semantically) obsolete messages from
the server without having to dequeue them.

> Another idea: provide a new exchange type, 'x-rabbitmq.last-value-topic', which would store the most recent message published for each topic in an Mnesia table.  During exchange.bind, the value in the cache would be retrieved and delivered to the queue immediately.  If no cached value was present, a special 'no value' message would be delivered instead.  Thoughts?  I am relatively new to both AMQP and RabbitMQ, so I'd appreciate any guidance here.

In terms of implementation, there may be a few ways to do this on the
server side:

- Maintain a queue per key and keep the max queue depth to 1 (crude,
but does the job)
- Maintain a priority search queue in the queue process that purges
the old value of a key when a new key arrives

In the second variant, you would somehow have to pass in the key to
the queue in a payload agnostic fashion - maybe you can use the
optional arguments parameter for publishing a message and (as you
suggest), use a special type of exchange.

One very simple way *could* be to use a hash map (you suggest mnesia,
which is a hash map on the heap) but you could equally just put stuff
into the process dictionary.

Having said that: how will you implement the FIFO semantics of the
queue? I don't know if a map will cut it in this scenario.

Also, what about message priorities and TTLs (which are not yet
implemented in Rabbit but I think that somebody in the community is
looking into this)?

BTW, what is the exchange.bind command?

I think that after considering all of these things, and investigating
some functional data structures, you will be able to come up with
something useful.

HTH,

Ben