[rabbitmq-discuss] internal sequence ID

Mon Jan 24 15:41:24 GMT 2011

On Mon, Jan 24, 2011 at 07:13:21AM -0600, Jon Brisbin wrote:
> I'm able to persist messages to Riak now but I'm trying to figure out how to return a proper sequence ID from publish/3.
> 
> What's the easiest way to implement this sequence ID that won't cause things to explode? :) Would an in-memory counter that starts at 1 all the time be sufficient? Or do I need to keep track of that by persisting some kind of metadata? It looks like variable_queue uses rabbit_queue_index, which I wasn't sure if I would be able to simply scab onto.

The seqid is a pure number. The purpose of queue_index is that at best,
without it, you'll end up with a queue of elements where each element
probably looks like {Guid, IsDelivered}. In order to avoid the memory
costs of queues of millions of such items, you need to be able to store
such queues on disk. If you assign each element a seqid then you'll be
able to compress down to what's helpfully called a δ in vq - i.e. a
start seqid, an end seqid and a count. And of course, there can be gaps
which is why you may need the count. At that point you can just lift off
disk the relevant records you need to be able to then rehydrate them to
{Guid, IsDelivered} elements, and from there get back to the whole msg.

How you achieve this in riak or elsewhere is of course entirely up to
you!

Matthew