[rabbitmq-discuss] How to handle full exchange history delivery

Fri Jun 3 00:14:11 BST 2011

I have started implementing it with a layer on top of rabbit, and it gets
tricky soon. There are two main challenges:

- figure out how to merge the stored and live message streams
- make sure that the replayer component replays messages which partially
overlap with those the client subscribed to before asking for stored
messages (this is a case of slow message store)

To solve the first I have to decorated the messages with a unique identifier
so that the client knows when to merge live and stored messages based on id
match.
The second one is a bit trickier and I didn't solve it yet. This happens if
the component which stores messages is slow and upon replay request is only
capable of replaying messages up to a point in time earlier than the first
message the client gets from the live stream receives, thus leaving a client
with a hole in the message stream.

On Thu, Jun 2, 2011 at 12:40, Michael Bridgen <mikeb at rabbitmq.com> wrote:

>  WRT this:
>>
>> /When I was writing it, I thought it would be really easy to, when a new
>> queue is bound to this exchange, go fetch all the messages in that
>> bucket and send them to the queue.///
>>
>> I’m not sure it is, that was my main question actually. How do you
>> figure out how to join the messages stored in the bucket and those
>> arriving live so that the last message you send to the queue from those
>> picked up from the bucket is followed exactly by its logical successor
>> from those coming live?
>>
>> Say I have messages 1..100 being published to the exchange, each time
>> one arrives it’s stored. The exchange has received the first 50 messages
>> and a client performs a binding of a queue to the exchange. So you start
>> pulling the messages from the storage and sending them to the queue
>> before forwarding any live messages. While you are pulling them out of
>> the storage messages 51..60 are published to the exchange, how do you
>> deal with them? Do you always pull from the storage? When do you stop
>> pulling from the storage and relay to the queue directly what is being
>> published to the exchange if you don’t know whether any new messages
>> have arrived while you were pulling from the storage?
>>
>
> My inclination with this and similar use cases* is that they deserve their
> own content class (alongside 'exchange', 'queue', etc.).
>
> By way of illustration: the last-value-caching exchange, though it was
> developed as a verification of the exchange type mechanism, is rather
> awkward; in the main because it is encoded in the existing AMQP operations
> (bind, consume), and these aren't the right vocabulary for the intended use.
>  See the "Limitations" section of
> https://github.com/squaremo/rabbitmq-lvc-plugin.
>
> There's two natural generalisations of LVC: one is to keep an arbitrary
> number of values around, and the other is to make the hold higher-order
> (i.e., instead of replaying values, replay some aggregate of values).
> Neither of these really fit in AMQP as it stands, not in the available
> methods or their semantics.
>
> Anyway.  So that this is helpful and not just me kicking AMQP, here is a
> suggestion: if the consumers are made to co-operate just slightly you can
> get your proxy idea to work. Just use different queues for the replay and
> the live.  Subscribe the live first and note the most recent message you
> know about at that time; then publish your saved messages through to that
> message into the replay queue.  The consumer, when it hears back from the
> proxy, drains the replay queue then starts on the live queue.
>
>
> Michael
>
> *I call the category "value+updates", and the synchronisation problem you
> describe above is the essence of it.
>
>  _______________________________________________________________
>>
>> *Simone Busoli*
>>
>> Direzione risorse umane e segreteria generale – Sistemi informativi
>> gestione sportiva
>> /Collaboratore/
>>
>> *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
>> *T* (+39) 0536-949.850
>> *E* _simone.busoli at ferrari.com <mailto:simone.busoli at ferrari.com>_
>> _www.ferrari.com <http://www.ferrari.com/>_
>>
>>
>> *From:* Jon Brisbin [mailto:jon at jbrisbin.com]
>> *Sent:* Wednesday, June 01, 2011 4:59 PM
>> *To:* Busoli, Simone
>> *Cc:* rabbitmq-discuss at lists.rabbitmq.com
>> *Subject:* Re: [rabbitmq-discuss] How to handle full exchange history
>> delivery
>>
>> Maybe I wasn't being clear:
>>
>> I've already written a custom exchange that stores every message it
>> receives in Riak (http://wiki.basho.com/). I'm going to be talking about
>> it next week at Erlang Factory in London, as a matter of fact. :)
>>
>> https://github.com/jbrisbin/riak-exchange
>>
>> When I was writing it, I thought it would be really easy to, when a new
>> queue is bound to this exchange, go fetch all the messages in that
>> bucket and send them to the queue.
>>
>> Since I'm using Riak, the messages themselves are not all stored in
>> memory.
>>
>> Thanks!
>>
>> Jon Brisbin
>> http//jbrisbin.com
>>
>> ------------------------------------------------------------------------
>>
>>    *From: *"Simone Busoli" <Simone.Busoli at ferrari.com>
>>    *To: *rabbitmq-discuss at lists.rabbitmq.com
>>    *Sent: *Wednesday, June 1, 2011 8:43:58 AM
>>    *Subject: *Re: [rabbitmq-discuss] How to handle full exchange
>>    history delivery
>>
>>
>>    That’s what I thought as well, although I’m thinking about a couple
>>    of issues about it:
>>
>>    · Lack of knowledge about Erlang, which is the language I’d have to
>>    use to build a custom exchange, I guess
>>
>>    · Amount of memory it would require to store the entire history in
>>    memory. It needn’t be in memory actually, but I’m not sure what
>>    would be the recommended way if the storage has to be carried out by
>>    exchange itself. Mnesia perhaps, would it make sense?
>>
>>    _______________________________________________________________
>>
>>    *Simone Busoli*
>>    Direzione risorse umane e segreteria generale – Sistemi informativi
>>    gestione sportiva
>>    /Collaboratore/
>>
>>    *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
>>    *T* (+39) 0536-949.850
>>    *E* _simone.busoli at ferrari.com <mailto:simone.busoli at ferrari.com>_
>>    _www.ferrari.com <http://www.ferrari.com/>_
>>
>>
>>    *From:* Jon Brisbin [mailto:jon at jbrisbin.com]
>>    *Sent:* Wednesday, June 01, 2011 3:38 PM
>>    *To:* Busoli, Simone
>>    *Cc:* rabbitmq-discuss at lists.rabbitmq.com
>>    *Subject:* Re: [rabbitmq-discuss] How to handle full exchange
>>    history delivery
>>
>>    I've been seriously considering adding this functionality to the
>>    Riak-backed custom exchange. It was something I thought I would add
>>    first time through but didn't.
>>
>>    You'd have to use a custom exchange. Not sure if that's an issue in
>>    your scenario or not.
>>
>>
>>    Thanks!
>>
>>    Jon Brisbin
>>    http//jbrisbin.com
>>
>>
>>  ------------------------------------------------------------------------
>>
>>        *From: *"Simone Busoli" <Simone.Busoli at ferrari.com>
>>        *To: *rabbitmq-discuss at lists.rabbitmq.com
>>        *Sent: *Wednesday, June 1, 2011 5:52:24 AM
>>        *Subject: *[rabbitmq-discuss] How to handle full exchange
>>        history delivery
>>
>>        Hi,
>>
>>        I am trying to achieve a scenario in which messages are
>>        published from some data source into a RabbitMQ exchange and
>>        clients should be able to receive the full history of messages
>>        flown through that exchange, regardless of the exact moment in
>>        time when they connect.
>>
>>        I’ve been thinking to solve it by setting up a proxy which
>>        subscribes to the exchange where messages are published using
>>        its own queue, keeps an internal storage of the messages and
>>        accepts requests from clients.
>>
>>        When a request from a client arrives it just replays the full
>>        message history to the client by publishing the messages to the
>>        client queue (supplied in the ReplyTo field), *then* binds the
>>        client queue to the messages exchange, so that the client from
>>        this moment onwards starts receiving the “live” messages, and is
>>        pretty much unaware of what’s happened; it simply just receives
>>        the whole history.
>>
>>        Now I am under the impression that there’s a fundamental flaw in
>>        this reasoning, since the proxy cannot know when the message
>>        history has completed and he can switch the client to the live
>>        stream, because while it is replaying historical messages to the
>>        client other live messages might be arriving on the proxy’s
>>        queue, thus getting into an endless loop.
>>
>>        I’m probably missing something obvious, any advice about how
>>        others would approach this scenario is appreciated.
>>
>>        Simone
>>
>>
>>  _________________________________________________________________________________________________________________
>>        Questo messaggio e da intendersi esclusivamente ad uso del
>>        destinatario e puo contenere informazioni che sono di natura
>>        privilegiata, confidenziale
>>        o non divulgabile secondo le leggi vigenti. Se il lettore del
>>        presente messaggio non e il destinatario designato, o il
>>        dipendente/agente responsabile
>>        per la consegna del messaggio al destinatario designato, si
>>        informa che ogni disseminazione, distribuzione o copiatura di
>>        questa comunicazione e
>>        strettamente proibita anche ai sensi del decreto legislativo
>>        196/03 . Se avete ricevuto questo messaggio per errore, vi
>>        preghiamo di notificarcelo
>>        immediatamente a mezzo e-mail di risposta e successivamente di
>>        procedere alla cancellazione di questa e-mail e relativi
>>        allegati dal vostro sistema.
>>
>>  _________________________________________________________________________________________________________________
>>        This message is intended only for the use of the addressee and
>>        may contain information that is privileged, confidential and
>>        exempt from
>>        disclosure under applicable law. If the reader of this message
>>        is not the intended recipient, or the employee or agent
>>        responsible for delivering the
>>        message to the intended recipient, you are hereby notified that
>>        any dissemination, distribution or copying of this communication
>>        is strictly
>>        prohibited. If you have received this e-mail in error, please
>>        notify us immediately by return e-mail and delete this e-mail
>>        and all attachments from
>>        your system.
>>
>>  _________________________________________________________________________________________________________________
>>
>>
>>        _______________________________________________
>>        rabbitmq-discuss mailing list
>>        rabbitmq-discuss at lists.rabbitmq.com
>>        <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>
>>
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>>  _________________________________________________________________________________________________________________
>>    Questo messaggio è da intendersi esclusivamente ad uso del
>>    destinatario e può contenere informazioni che sono di natura
>>    privilegiata, confidenziale
>>    o non divulgabile secondo le leggi vigenti. Se il lettore del
>>    presente messaggio non è il destinatario designato, o il
>>    dipendente/agente responsabile
>>    per la consegna del messaggio al destinatario designato, si informa
>>    che ogni disseminazione, distribuzione o copiatura di questa
>>    comunicazione è
>>    strettamente proibita anche ai sensi del decreto legislativo 196/03
>>    . Se avete ricevuto questo messaggio per errore, vi preghiamo di
>>    notificarcelo
>>    immediatamente a mezzo e-mail di risposta e successivamente di
>>    procedere alla cancellazione di questa e-mail e relativi allegati
>>    dal vostro sistema.
>>
>>  _________________________________________________________________________________________________________________
>>    This message is intended only for the use of the addressee and may
>>    contain information that is privileged, confidential and exempt from
>>    disclosure under applicable law. If the reader of this message is
>>    not the intended recipient, or the employee or agent responsible for
>>    delivering the
>>    message to the intended recipient, you are hereby notified that any
>>    dissemination, distribution or copying of this communication is
>>    strictly
>>    prohibited. If you have received this e-mail in error, please notify
>>    us immediately by return e-mail and delete this e-mail and all
>>    attachments from
>>    your system.
>>
>>  _________________________________________________________________________________________________________________
>>
>>
>>    _______________________________________________
>>    rabbitmq-discuss mailing list
>>    rabbitmq-discuss at lists.rabbitmq.com
>>    <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>
>>    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio è da intendersi esclusivamente ad uso del destinatario
>> e può contenere informazioni che sono di natura privilegiata,
>> confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del presente
>> messaggio non è il destinatario designato, o il dipendente/agente
>> responsabile
>> per la consegna del messaggio al destinatario designato, si informa che
>> ogni disseminazione, distribuzione o copiatura di questa comunicazione è
>> strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se
>> avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di procedere
>> alla cancellazione di questa e-mail e relativi allegati dal vostro
>> sistema.
>>
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is not
>> the intended recipient, or the employee or agent responsible for
>> delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify us
>> immediately by return e-mail and delete this e-mail and all attachments
>> from
>> your system.
>>
>> _________________________________________________________________________________________________________________
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110603/9bbf59eb/attachment-0001.htm>