[rabbitmq-discuss] is it possible to receive old messages on subscribe ?

[Ben Hood]

Jérémy,

On Thu, Sep 3, 2009 at 10:50 AM, Jérémy Lal<kapouer at melix.org> wrote:
> Now for some queue, i'd like to make every new subscriber receive all messages
> previously sent to the queue (by some privileged user).
>
> Is there a way to do this ? if yes, where do i need to look at ?
> i tried reading the AMQP spec, and did not understand it well enough to find
> an answer...

I think the only clean way to do this on a pure AMQP level is to have
create and bind queues for consumers that don't yet exist - so that
they will get a *full* queue when they subscribe - they will consume
all of the previously sent messages first and then continue with the
stuff that trickles in afterwards. Since queue paging is only landing
in the next Rabbit release, this *could* cost you a lot of memory with
the current version. Also, you would have to know about all new
consumers ahead of time.

Conceptually, I think you need to ask yourself whether you are turning
your messaging system into a database. If you are, then maybe it's not
such a good idea to implement a database in Rabbit.

Having said this, this is a reasonably common pattern - to implement
this, some people have set up routing rules to archive each message in
addition to normal consumers subscribing to queues - this routing is
the value-add of AMQP. Then in their application they implement the
subscription logic as follows:

- Create and bind a queue to the to real time exchange, but don't
subscribe yet, just let messages queue up;
- Send a request to the archive to replay archived messages;
- Process the archived messages;
- Subscribe to the queue you bound in step one and handle the initial
duplicates;
- Continue consuming the trickle feed as normal;

There are a few improvements some people have made to this kind of thing:

- Implement monotonically increasing ids for your messages so that you
can replay from an arbitrary point;
- Consider a non-AMQP bulk transfer mechanism for recovering old
messages , e.g. HTTP - this can be useful if your consumers
potentially have to recover a vast amount of old data;

HTH,

Ben