[rabbitmq-discuss] How to handle full exchange history delivery

Thu Jun 2 14:36:13 BST 2011

Thanks Jon, I see your point. I'm really just evaluating ways to accomplish
this scenario and went down to trying to carry it out by relying on RabbitMQ
alone, which I get doesn't fit very well here.
I will probably fallback to creating a layer on top of it which takes care
of message ordering and merging of storage and live streams. Thanks for
sharing your ideas!
On Jun 1, 2011 5:51 PM, "Jon Brisbin" <jon at jbrisbin.com> wrote:
> I'm not a big fan of forcing the message broker to keep fascist control
over message ordering. Spring Integration has great tools for handling
splitting and aggregation that can do this message ordering for you, which
means your message broker situation can be much more flexible.
>
> In this particular case, the only way to guarantee you won't receive new
messages before you've finished sending them the old ones is to suspend
message delivery of new messages to the newly-bound queue until it is
finished sending the old ones. One would have to literally queue requests to
route messages internally (to the exchange). The problem is that exchanges
are stateless. One would have to actually store new message routing requests
in mnesia or something based on whether or not the currently-being-filled
queue has a new routing request pending.
>
> All that said, it's technically possible to implement this kind of
behaviour in the broker. I'm just not sure it would be a good idea. :)
Messaging applications are, after all, supposed to be "A"synchronous, which
means, necessarily, undetermined ordering. If ordering is really that
important, then you don't want a message broker, you want a datastore and a
cursor of some sort to sequentially process ordered items. You need, in
fact, a datastore just like the one I'm currently proof-of-concepting (is
that a word? ;) in my spare time, which is an asynchronous datastore that
mixes messaging and data storage to achieve very closely to what you're
wanting here. ;)
>
> Thanks!
>
> Jon Brisbin
> http//jbrisbin.com
>
> ----- Original Message -----
>
>> From: "Simone Busoli" <Simone.Busoli at ferrari.com>
>> To: rabbitmq-discuss at lists.rabbitmq.com
>> Sent: Wednesday, June 1, 2011 10:15:26 AM
>> Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> delivery
>
>> Thanks Jon, I misunderstood you probably.
>
>> WRT this:
>
>> When I was writing it, I thought it would be really easy to, when a
>> new queue is bound to this exchange, go fetch all the messages in
>> that bucket and send them to the queue.
>
>> I’m not sure it is, that was my main question actually. How do you
>> figure out how to join the messages stored in the bucket and those
>> arriving live so that the last message you send to the queue from
>> those picked up from the bucket is followed exactly by its logical
>> successor from those coming live?
>
>> Say I have messages 1..100 being published to the exchange, each time
>> one arrives it’s stored. The exchange has received the first 50
>> messages and a client performs a binding of a queue to the exchange.
>> So you start pulling the messages from the storage and sending them
>> to the queue before forwarding any live messages. While you are
>> pulling them out of the storage messages 51..60 are published to the
>> exchange, how do you deal with them? Do you always pull from the
>> storage? When do you stop pulling from the storage and relay to the
>> queue directly what is being published to the exchange if you don’t
>> know whether any new messages have arrived while you were pulling
>> from the storage?
>
>> _______________________________________________________________
>> Simone Busoli
>> Direzione risorse umane e segreteria generale – Sistemi informativi
>> gestione sportiva
>> Collaboratore
>> Ferrari S.p.A. Via Ascari 55/57, 41053, Maranello (MO); Italy
>> T (+39) 0536-949.850
>> E simone.busoli at ferrari.com www.ferrari.com
>
>> From: Jon Brisbin [mailto:jon at jbrisbin.com]
>> Sent: Wednesday, June 01, 2011 4:59 PM
>> To: Busoli, Simone
>> Cc: rabbitmq-discuss at lists.rabbitmq.com
>> Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> delivery
>
>> Maybe I wasn't being clear:
>
>> I've already written a custom exchange that stores every message it
>> receives in Riak ( http://wiki.basho.com/ ). I'm going to be talking
>> about it next week at Erlang Factory in London, as a matter of fact.
>> :)
>
>> https://github.com/jbrisbin/riak-exchange
>
>> When I was writing it, I thought it would be really easy to, when a
>> new queue is bound to this exchange, go fetch all the messages in
>> that bucket and send them to the queue.
>
>> Since I'm using Riak, the messages themselves are not all stored in
>> memory.
>
>> Thanks!
>
>> Jon Brisbin
>> http//jbrisbin.com
>
>> ----- Original Message -----
>
>> > From: "Simone Busoli" <Simone.Busoli at ferrari.com>
>>
>> > To: rabbitmq-discuss at lists.rabbitmq.com
>>
>> > Sent: Wednesday, June 1, 2011 8:43:58 AM
>>
>> > Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> > delivery
>>
>
>> > That’s what I thought as well, although I’m thinking about a couple
>> > of issues about it:
>>
>
>> > · Lack of knowledge about Erlang, which is the language I’d have to
>> > use to build a custom exchange, I guess
>>
>> > · Amount of memory it would require to store the entire history in
>> > memory. It needn’t be in memory actually, but I’m not sure what
>> > would be the recommended way if the storage has to be carried out
>> > by
>> > exchange itself. Mnesia perhaps, would it make sense?
>>
>
>> > _______________________________________________________ ________
>>
>> > Simone Busoli
>>
>> > Direzione risorse umane e segreteria generale – Sistemi informativi
>> > gestione sportiva
>>
>> > Collaboratore
>>
>> > Ferrari S.p.A. Via Ascari 55/57, 41053, Maranello (MO); Italy
>>
>> > T (+39) 0536-949.850
>>
>> > E simone.busoli at ferrari.com www.ferrari.com
>>
>
>> > From: Jon Brisbin [mailto:jon at jbrisbin.com]
>>
>> > Sent: Wednesday, June 01, 2011 3:38 PM
>>
>> > To: Busoli, Simone
>>
>> > Cc: rabbitmq-discuss at lists.rabbitmq.com
>>
>> > Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> > delivery
>>
>
>> > I've been seriously considering adding this functionality to the
>> > Riak-backed custom exchange. It was something I thought I would add
>> > first time through but didn't.
>>
>
>> > You'd have to use a custom exchange. Not sure if that's an issue in
>> > your scenario or not.
>>
>
>> > Thanks!
>>
>
>> > Jon Brisbin
>>
>> > http//jbrisbin.com
>>
>
>> > > From: "Simone Busoli" <Simone.Busoli at ferrari.com>
>> >
>>
>> > > To: rabbitmq-discuss at lists.rabbitmq.com
>> >
>>
>> > > Sent: Wednesday, June 1, 2011 5:52:24 AM
>> >
>>
>> > > Subject: [rabbitmq-discuss] How to handle full exchange history
>> > > delivery
>> >
>>
>
>> > > Hi,
>> >
>>
>
>> > > I am trying to achieve a scenario in which messages are published
>> > > from some data source into a RabbitMQ exchange and clients should
>> > > be
>> > > able to receive the full history of messages flown through that
>> > > exchange, regardless of the exact moment in time when they
>> > > connect.
>> >
>>
>
>> > > I’ve been thinking to solve it by setting up a proxy which
>> > > subscribes
>> > > to the exchange where messages are published using its own queue,
>> > > keeps an internal storage of the messages and accepts requests
>> > > from
>> > > clients.
>> >
>>
>> > > When a request from a client arrives it just replays the full
>> > > message
>> > > history to the client by publishing the messages to the client
>> > > queue
>> > > (supplied in the ReplyTo field), then binds the client queue to
>> > > the
>> > > messages exchange, so that the client from this moment onwards
>> > > starts receiving the “live” messages, and is pretty much unaware
>> > > of
>> > > what’s happened; it simply just receives the whole history.
>> >
>>
>
>> > > Now I am under the impression that there’s a fundamental flaw in
>> > > this
>> > > reasoning, since the proxy cannot know when the message history
>> > > has
>> > > completed and he can switch the client to the live stream,
>> > > because
>> > > while it is replaying historical messages to the client other
>> > > live
>> > > messages might be arriving on the proxy’s queue, thus getting
>> > > into
>> > > an endless loop.
>> >
>>
>
>> > > I’m probably missing something obvious, any advice about how
>> > > others
>> > > would approach this scenario is appreciated.
>> >
>>
>
>> > > Simone
>> >
>>
>> > >
_________________________________________________________________________________________________________________
>> >
>>
>> > > Questo messaggio e da intendersi esclusivamente ad uso del
>> > > destinatario e puo contenere informazioni che sono di natura
>> > > privilegiata, confidenziale
>> >
>>
>> > > o non divulgabile secondo le leggi vigenti. Se il lettore del
>> > > presente messaggio non e il destinatario designato, o il
>> > > dipendente/agente responsabile
>> >
>>
>> > > per la consegna del messaggio al destinatario designato, si
>> > > informa
>> > > che ogni disseminazione, distribuzione o copiatura di questa
>> > > comunicazione e
>> >
>>
>> > > strettamente proibita anche ai sensi del decreto legislativo
>> > > 196/03
>> > > .
>> > > Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> > > notificarcelo
>> >
>>
>> > > immediatamente a mezzo e-mail di risposta e successivamente di
>> > > procedere alla cancellazione di questa e-mail e relativi allegati
>> > > dal vostro sistema.
>> >
>>
>> > >
_________________________________________________________________________________________________________________
>> >
>>
>> > > This message is intended only for the use of the addressee and
>> > > may
>> > > contain information that is privileged, confidential and exempt
>> > > from
>> >
>>
>> > > disclosure under applicable law. If the reader of this message is
>> > > not
>> > > the intended recipient, or the employee or agent responsible for
>> > > delivering the
>> >
>>
>> > > message to the intended recipient, you are hereby notified that
>> > > any
>> > > dissemination, distribution or copying of this communication is
>> > > strictly
>> >
>>
>> > > prohibited. If you have received this e-mail in error, please
>> > > notify
>> > > us immediately by return e-mail and delete this e-mail and all
>> > > attachments from
>> >
>>
>> > > your system.
>> >
>>
>> > >
_________________________________________________________________________________________________________________
>> >
>>
>
>> > > _______________________________________________
>> >
>>
>> > > rabbitmq-discuss mailing list
>> >
>>
>> > > rabbitmq-discuss at lists.rabbitmq.com
>> >
>>
>> > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>>
>> >
_________________________________________________________________________________________________________________
>>
>> > Questo messaggio è da intendersi esclusivamente ad uso del
>> > destinatario e può contenere informazioni che sono di natura
>> > privilegiata, confidenziale
>>
>> > o non divulgabile secondo le leggi vigenti. Se il lettore del
>> > presente messaggio non è il destinatario designato, o il
>> > dipendente/agente responsabile
>>
>> > per la consegna del messaggio al destinatario designato, si informa
>> > che ogni disseminazione, distribuzione o copiatura di questa
>> > comunicazione è
>>
>> > strettamente proibita anche ai sensi del decreto legislativo 196/03
>> > .
>> > Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> > notificarcelo
>>
>> > immediatamente a mezzo e-mail di risposta e successivamente di
>> > procedere alla cancellazione di questa e-mail e relativi allegati
>> > dal vostro sistema.
>>
>> >
_________________________________________________________________________________________________________________
>>
>> > This message is intended only for the use of the addressee and may
>> > contain information that is privileged, confidential and exempt
>> > from
>>
>> > disclosure under applicable law. If the reader of this message is
>> > not
>> > the intended recipient, or the employee or agent responsible for
>> > delivering the
>>
>> > message to the intended recipient, you are hereby notified that any
>> > dissemination, distribution or copying of this communication is
>> > strictly
>>
>> > prohibited. If you have received this e-mail in error, please
>> > notify
>> > us immediately by return e-mail and delete this e-mail and all
>> > attachments from
>>
>> > your system.
>>
>> >
_________________________________________________________________________________________________________________
>>
>
>> > _______________________________________________
>>
>> > rabbitmq-discuss mailing list
>>
>> > rabbitmq-discuss at lists.rabbitmq.com
>>
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
_________________________________________________________________________________________________________________
>> Questo messaggio è da intendersi esclusivamente ad uso del
>> destinatario e può contenere informazioni che sono di natura
>> privilegiata, confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del
>> presente messaggio non è il destinatario designato, o il
>> dipendente/agente responsabile
>> per la consegna del messaggio al destinatario designato, si informa
>> che ogni disseminazione, distribuzione o copiatura di questa
>> comunicazione è
>> strettamente proibita anche ai sensi del decreto legislativo 196/03 .
>> Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di
>> procedere alla cancellazione di questa e-mail e relativi allegati
>> dal vostro sistema.
>>
_________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is not
>> the intended recipient, or the employee or agent responsible for
>> delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is
>> strictly
>> prohibited. If you have received this e-mail in error, please notify
>> us immediately by return e-mail and delete this e-mail and all
>> attachments from
>> your system.
>>
_________________________________________________________________________________________________________________
>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110602/c0f28dba/attachment-0001.htm>