[rabbitmq-discuss] How to handle full exchange history delivery
Simone Busoli
simone.busoli at gmail.com
Fri Jun 3 00:14:11 BST 2011
I have started implementing it with a layer on top of rabbit, and it gets
tricky soon. There are two main challenges:
- figure out how to merge the stored and live message streams
- make sure that the replayer component replays messages which partially
overlap with those the client subscribed to before asking for stored
messages (this is a case of slow message store)
To solve the first I have to decorated the messages with a unique identifier
so that the client knows when to merge live and stored messages based on id
match.
The second one is a bit trickier and I didn't solve it yet. This happens if
the component which stores messages is slow and upon replay request is only
capable of replaying messages up to a point in time earlier than the first
message the client gets from the live stream receives, thus leaving a client
with a hole in the message stream.
On Thu, Jun 2, 2011 at 12:40, Michael Bridgen <mikeb at rabbitmq.com> wrote:
> WRT this:
>>
>> /When I was writing it, I thought it would be really easy to, when a new
>> queue is bound to this exchange, go fetch all the messages in that
>> bucket and send them to the queue.///
>>
>> I’m not sure it is, that was my main question actually. How do you
>> figure out how to join the messages stored in the bucket and those
>> arriving live so that the last message you send to the queue from those
>> picked up from the bucket is followed exactly by its logical successor
>> from those coming live?
>>
>> Say I have messages 1..100 being published to the exchange, each time
>> one arrives it’s stored. The exchange has received the first 50 messages
>> and a client performs a binding of a queue to the exchange. So you start
>> pulling the messages from the storage and sending them to the queue
>> before forwarding any live messages. While you are pulling them out of
>> the storage messages 51..60 are published to the exchange, how do you
>> deal with them? Do you always pull from the storage? When do you stop
>> pulling from the storage and relay to the queue directly what is being
>> published to the exchange if you don’t know whether any new messages
>> have arrived while you were pulling from the storage?
>>
>
> My inclination with this and similar use cases* is that they deserve their
> own content class (alongside 'exchange', 'queue', etc.).
>
> By way of illustration: the last-value-caching exchange, though it was
> developed as a verification of the exchange type mechanism, is rather
> awkward; in the main because it is encoded in the existing AMQP operations
> (bind, consume), and these aren't the right vocabulary for the intended use.
> See the "Limitations" section of
> https://github.com/squaremo/rabbitmq-lvc-plugin.
>
> There's two natural generalisations of LVC: one is to keep an arbitrary
> number of values around, and the other is to make the hold higher-order
> (i.e., instead of replaying values, replay some aggregate of values).
> Neither of these really fit in AMQP as it stands, not in the available
> methods or their semantics.
>
> Anyway. So that this is helpful and not just me kicking AMQP, here is a
> suggestion: if the consumers are made to co-operate just slightly you can
> get your proxy idea to work. Just use different queues for the replay and
> the live. Subscribe the live first and note the most recent message you
> know about at that time; then publish your saved messages through to that
> message into the replay queue. The consumer, when it hears back from the
> proxy, drains the replay queue then starts on the live queue.
>
>
> Michael
>
> *I call the category "value+updates", and the synchronisation problem you
> describe above is the essence of it.
>
> _______________________________________________________________
>>
>> *Simone Busoli*
>>
>> Direzione risorse umane e segreteria generale – Sistemi informativi
>> gestione sportiva
>> /Collaboratore/
>>
>> *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
>> *T* (+39) 0536-949.850
>> *E* _simone.busoli at ferrari.com <mailto:simone.busoli at ferrari.com>_
>> _www.ferrari.com <http://www.ferrari.com/>_
>>
>>
>> *From:* Jon Brisbin [mailto:jon at jbrisbin.com]
>> *Sent:* Wednesday, June 01, 2011 4:59 PM
>> *To:* Busoli, Simone
>> *Cc:* rabbitmq-discuss at lists.rabbitmq.com
>> *Subject:* Re: [rabbitmq-discuss] How to handle full exchange history
>> delivery
>>
>> Maybe I wasn't being clear:
>>
>> I've already written a custom exchange that stores every message it
>> receives in Riak (http://wiki.basho.com/). I'm going to be talking about
>> it next week at Erlang Factory in London, as a matter of fact. :)
>>
>> https://github.com/jbrisbin/riak-exchange
>>
>> When I was writing it, I thought it would be really easy to, when a new
>> queue is bound to this exchange, go fetch all the messages in that
>> bucket and send them to the queue.
>>
>> Since I'm using Riak, the messages themselves are not all stored in
>> memory.
>>
>> Thanks!
>>
>> Jon Brisbin
>> http//jbrisbin.com
>>
>> ------------------------------------------------------------------------
>>
>> *From: *"Simone Busoli" <Simone.Busoli at ferrari.com>
>> *To: *rabbitmq-discuss at lists.rabbitmq.com
>> *Sent: *Wednesday, June 1, 2011 8:43:58 AM
>> *Subject: *Re: [rabbitmq-discuss] How to handle full exchange
>> history delivery
>>
>>
>> That’s what I thought as well, although I’m thinking about a couple
>> of issues about it:
>>
>> · Lack of knowledge about Erlang, which is the language I’d have to
>> use to build a custom exchange, I guess
>>
>> · Amount of memory it would require to store the entire history in
>> memory. It needn’t be in memory actually, but I’m not sure what
>> would be the recommended way if the storage has to be carried out by
>> exchange itself. Mnesia perhaps, would it make sense?
>>
>> _______________________________________________________________
>>
>> *Simone Busoli*
>> Direzione risorse umane e segreteria generale – Sistemi informativi
>> gestione sportiva
>> /Collaboratore/
>>
>> *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
>> *T* (+39) 0536-949.850
>> *E* _simone.busoli at ferrari.com <mailto:simone.busoli at ferrari.com>_
>> _www.ferrari.com <http://www.ferrari.com/>_
>>
>>
>> *From:* Jon Brisbin [mailto:jon at jbrisbin.com]
>> *Sent:* Wednesday, June 01, 2011 3:38 PM
>> *To:* Busoli, Simone
>> *Cc:* rabbitmq-discuss at lists.rabbitmq.com
>> *Subject:* Re: [rabbitmq-discuss] How to handle full exchange
>> history delivery
>>
>> I've been seriously considering adding this functionality to the
>> Riak-backed custom exchange. It was something I thought I would add
>> first time through but didn't.
>>
>> You'd have to use a custom exchange. Not sure if that's an issue in
>> your scenario or not.
>>
>>
>> Thanks!
>>
>> Jon Brisbin
>> http//jbrisbin.com
>>
>>
>> ------------------------------------------------------------------------
>>
>> *From: *"Simone Busoli" <Simone.Busoli at ferrari.com>
>> *To: *rabbitmq-discuss at lists.rabbitmq.com
>> *Sent: *Wednesday, June 1, 2011 5:52:24 AM
>> *Subject: *[rabbitmq-discuss] How to handle full exchange
>> history delivery
>>
>> Hi,
>>
>> I am trying to achieve a scenario in which messages are
>> published from some data source into a RabbitMQ exchange and
>> clients should be able to receive the full history of messages
>> flown through that exchange, regardless of the exact moment in
>> time when they connect.
>>
>> I’ve been thinking to solve it by setting up a proxy which
>> subscribes to the exchange where messages are published using
>> its own queue, keeps an internal storage of the messages and
>> accepts requests from clients.
>>
>> When a request from a client arrives it just replays the full
>> message history to the client by publishing the messages to the
>> client queue (supplied in the ReplyTo field), *then* binds the
>> client queue to the messages exchange, so that the client from
>> this moment onwards starts receiving the “live” messages, and is
>> pretty much unaware of what’s happened; it simply just receives
>> the whole history.
>>
>> Now I am under the impression that there’s a fundamental flaw in
>> this reasoning, since the proxy cannot know when the message
>> history has completed and he can switch the client to the live
>> stream, because while it is replaying historical messages to the
>> client other live messages might be arriving on the proxy’s
>> queue, thus getting into an endless loop.
>>
>> I’m probably missing something obvious, any advice about how
>> others would approach this scenario is appreciated.
>>
>> Simone
>>
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio e da intendersi esclusivamente ad uso del
>> destinatario e puo contenere informazioni che sono di natura
>> privilegiata, confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del
>> presente messaggio non e il destinatario designato, o il
>> dipendente/agente responsabile
>> per la consegna del messaggio al destinatario designato, si
>> informa che ogni disseminazione, distribuzione o copiatura di
>> questa comunicazione e
>> strettamente proibita anche ai sensi del decreto legislativo
>> 196/03 . Se avete ricevuto questo messaggio per errore, vi
>> preghiamo di notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di
>> procedere alla cancellazione di questa e-mail e relativi
>> allegati dal vostro sistema.
>>
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and
>> may contain information that is privileged, confidential and
>> exempt from
>> disclosure under applicable law. If the reader of this message
>> is not the intended recipient, or the employee or agent
>> responsible for delivering the
>> message to the intended recipient, you are hereby notified that
>> any dissemination, distribution or copying of this communication
>> is strictly
>> prohibited. If you have received this e-mail in error, please
>> notify us immediately by return e-mail and delete this e-mail
>> and all attachments from
>> your system.
>>
>> _________________________________________________________________________________________________________________
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>
>>
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio è da intendersi esclusivamente ad uso del
>> destinatario e può contenere informazioni che sono di natura
>> privilegiata, confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del
>> presente messaggio non è il destinatario designato, o il
>> dipendente/agente responsabile
>> per la consegna del messaggio al destinatario designato, si informa
>> che ogni disseminazione, distribuzione o copiatura di questa
>> comunicazione è
>> strettamente proibita anche ai sensi del decreto legislativo 196/03
>> . Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di
>> procedere alla cancellazione di questa e-mail e relativi allegati
>> dal vostro sistema.
>>
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is
>> not the intended recipient, or the employee or agent responsible for
>> delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is
>> strictly
>> prohibited. If you have received this e-mail in error, please notify
>> us immediately by return e-mail and delete this e-mail and all
>> attachments from
>> your system.
>>
>> _________________________________________________________________________________________________________________
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio è da intendersi esclusivamente ad uso del destinatario
>> e può contenere informazioni che sono di natura privilegiata,
>> confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del presente
>> messaggio non è il destinatario designato, o il dipendente/agente
>> responsabile
>> per la consegna del messaggio al destinatario designato, si informa che
>> ogni disseminazione, distribuzione o copiatura di questa comunicazione è
>> strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se
>> avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di procedere
>> alla cancellazione di questa e-mail e relativi allegati dal vostro
>> sistema.
>>
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is not
>> the intended recipient, or the employee or agent responsible for
>> delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify us
>> immediately by return e-mail and delete this e-mail and all attachments
>> from
>> your system.
>>
>> _________________________________________________________________________________________________________________
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110603/9bbf59eb/attachment-0001.htm>
More information about the rabbitmq-discuss
mailing list