<font face="trebuchet ms,sans-serif">I have started implementing it with a layer on top of rabbit, and it gets tricky soon. There are two main challenges:</font><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><br>
</font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">- figure out how to merge the stored and live message streams</font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">- make sure that the replayer component replays messages which partially overlap with those the client subscribed to before asking for stored messages (this is a case of slow message store)</font></div>
<div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><br></font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">To solve the first I have to decorated the messages with a unique identifier so that the client knows when to merge live and stored messages based on id match.</font></div>
<div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">The second one is a bit trickier and I didn't solve it yet. This happens if the component which stores messages is slow and upon replay request is only capable of replaying messages up to a point in time earlier than the first message the client gets from the live stream receives, thus leaving a client with a hole in the message stream.</font></div>
<div><div><br><div class="gmail_quote">On Thu, Jun 2, 2011 at 12:40, Michael Bridgen <span dir="ltr"><<a href="mailto:mikeb@rabbitmq.com">mikeb@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
WRT this:<br>
<br>
/When I was writing it, I thought it would be really easy to, when a new<br>
queue is bound to this exchange, go fetch all the messages in that<br>
bucket and send them to the queue.///<br>
<br>
I�m not sure it is, that was my main question actually. How do you<br>
figure out how to join the messages stored in the bucket and those<br>
arriving live so that the last message you send to the queue from those<br>
picked up from the bucket is followed exactly by its logical successor<br>
from those coming live?<br>
<br>
Say I have messages 1..100 being published to the exchange, each time<br>
one arrives it�s stored. The exchange has received the first 50 messages<br>
and a client performs a binding of a queue to the exchange. So you start<br>
pulling the messages from the storage and sending them to the queue<br>
before forwarding any live messages. While you are pulling them out of<br>
the storage messages 51..60 are published to the exchange, how do you<br>
deal with them? Do you always pull from the storage? When do you stop<br>
pulling from the storage and relay to the queue directly what is being<br>
published to the exchange if you don�t know whether any new messages<br>
have arrived while you were pulling from the storage?<br>
</blockquote>
<br></div>
My inclination with this and similar use cases* is that they deserve their own content class (alongside 'exchange', 'queue', etc.).<br>
<br>
By way of illustration: the last-value-caching exchange, though it was developed as a verification of the exchange type mechanism, is rather awkward; in the main because it is encoded in the existing AMQP operations (bind, consume), and these aren't the right vocabulary for the intended use. �See the "Limitations" section of<br>
<a href="https://github.com/squaremo/rabbitmq-lvc-plugin" target="_blank">https://github.com/squaremo/rabbitmq-lvc-plugin</a>.<br>
<br>
There's two natural generalisations of LVC: one is to keep an arbitrary number of values around, and the other is to make the hold higher-order (i.e., instead of replaying values, replay some aggregate of values). Neither of these really fit in AMQP as it stands, not in the available methods or their semantics.<br>
<br>
Anyway. �So that this is helpful and not just me kicking AMQP, here is a suggestion: if the consumers are made to co-operate just slightly you can get your proxy idea to work. Just use different queues for the replay and the live. �Subscribe the live first and note the most recent message you know about at that time; then publish your saved messages through to that message into the replay queue. �The consumer, when it hears back from the proxy, drains the replay queue then starts on the live queue.<br>
<br>
<br>
Michael<br>
<br>
*I call the category "value+updates", and the synchronisation problem you describe above is the essence of it.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
_______________________________________________________________<br>
<br>
*Simone Busoli*<div class="im"><br>
Direzione risorse umane e segreteria generale � Sistemi informativi<br>
gestione sportiva<br>
/Collaboratore/<br>
<br>
*Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>
*T* <a href="tel:%28%2B39%29%200536-949.850" value="+390536949850" target="_blank">(+39) 0536-949.850</a><br></div>
*E* _<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a> <mailto:<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a>>_<br>
_<a href="http://www.ferrari.com" target="_blank">www.ferrari.com</a> <<a href="http://www.ferrari.com/" target="_blank">http://www.ferrari.com/</a>>_<div><div></div><div class="h5"><br>
<br>
*From:* Jon Brisbin [mailto:<a href="mailto:jon@jbrisbin.com" target="_blank">jon@jbrisbin.com</a>]<br>
*Sent:* Wednesday, June 01, 2011 4:59 PM<br>
*To:* Busoli, Simone<br>
*Cc:* <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
*Subject:* Re: [rabbitmq-discuss] How to handle full exchange history<br>
delivery<br>
<br>
Maybe I wasn't being clear:<br>
<br>
I've already written a custom exchange that stores every message it<br>
receives in Riak (<a href="http://wiki.basho.com/" target="_blank">http://wiki.basho.com/</a>). I'm going to be talking about<br>
it next week at Erlang Factory in London, as a matter of fact. :)<br>
<br>
<a href="https://github.com/jbrisbin/riak-exchange" target="_blank">https://github.com/jbrisbin/riak-exchange</a><br>
<br>
When I was writing it, I thought it would be really easy to, when a new<br>
queue is bound to this exchange, go fetch all the messages in that<br>
bucket and send them to the queue.<br>
<br>
Since I'm using Riak, the messages themselves are not all stored in memory.<br>
<br>
Thanks!<br>
<br>
Jon Brisbin<br>
http//<a href="http://jbrisbin.com" target="_blank">jbrisbin.com</a><br>
<br>
------------------------------------------------------------------------<br>
<br>
� �*From: *"Simone Busoli" <<a href="mailto:Simone.Busoli@ferrari.com" target="_blank">Simone.Busoli@ferrari.com</a>><br>
� �*To: *<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
� �*Sent: *Wednesday, June 1, 2011 8:43:58 AM<br>
� �*Subject: *Re: [rabbitmq-discuss] How to handle full exchange<br>
� �history delivery<br>
<br>
<br>
� �That�s what I thought as well, although I�m thinking about a couple<br>
� �of issues about it:<br>
<br>
� �� Lack of knowledge about Erlang, which is the language I�d have to<br>
� �use to build a custom exchange, I guess<br>
<br>
� �� Amount of memory it would require to store the entire history in<br>
� �memory. It needn�t be in memory actually, but I�m not sure what<br>
� �would be the recommended way if the storage has to be carried out by<br>
� �exchange itself. Mnesia perhaps, would it make sense?<br>
<br>
� �_______________________________________________________________<br>
<br>
� �*Simone Busoli*<br>
� �Direzione risorse umane e segreteria generale � Sistemi informativi<br>
� �gestione sportiva<br>
� �/Collaboratore/<br>
<br>
� �*Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>
� �*T* <a href="tel:%28%2B39%29%200536-949.850" value="+390536949850" target="_blank">(+39) 0536-949.850</a><br></div></div>
� �*E* _<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a> <mailto:<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a>>_<br>
� �_<a href="http://www.ferrari.com" target="_blank">www.ferrari.com</a> <<a href="http://www.ferrari.com/" target="_blank">http://www.ferrari.com/</a>>_<div><div></div><div class="h5"><br>
<br>
� �*From:* Jon Brisbin [mailto:<a href="mailto:jon@jbrisbin.com" target="_blank">jon@jbrisbin.com</a>]<br>
� �*Sent:* Wednesday, June 01, 2011 3:38 PM<br>
� �*To:* Busoli, Simone<br>
� �*Cc:* <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
� �*Subject:* Re: [rabbitmq-discuss] How to handle full exchange<br>
� �history delivery<br>
<br>
� �I've been seriously considering adding this functionality to the<br>
� �Riak-backed custom exchange. It was something I thought I would add<br>
� �first time through but didn't.<br>
<br>
� �You'd have to use a custom exchange. Not sure if that's an issue in<br>
� �your scenario or not.<br>
<br>
<br>
� �Thanks!<br>
<br>
� �Jon Brisbin<br>
� �http//<a href="http://jbrisbin.com" target="_blank">jbrisbin.com</a><br>
<br>
� �------------------------------------------------------------------------<br>
<br>
� � � �*From: *"Simone Busoli" <<a href="mailto:Simone.Busoli@ferrari.com" target="_blank">Simone.Busoli@ferrari.com</a>><br>
� � � �*To: *<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
� � � �*Sent: *Wednesday, June 1, 2011 5:52:24 AM<br>
� � � �*Subject: *[rabbitmq-discuss] How to handle full exchange<br>
� � � �history delivery<br>
<br>
� � � �Hi,<br>
<br>
� � � �I am trying to achieve a scenario in which messages are<br>
� � � �published from some data source into a RabbitMQ exchange and<br>
� � � �clients should be able to receive the full history of messages<br>
� � � �flown through that exchange, regardless of the exact moment in<br>
� � � �time when they connect.<br>
<br>
� � � �I�ve been thinking to solve it by setting up a proxy which<br>
� � � �subscribes to the exchange where messages are published using<br>
� � � �its own queue, keeps an internal storage of the messages and<br>
� � � �accepts requests from clients.<br>
<br>
� � � �When a request from a client arrives it just replays the full<br>
� � � �message history to the client by publishing the messages to the<br>
� � � �client queue (supplied in the ReplyTo field), *then* binds the<br>
� � � �client queue to the messages exchange, so that the client from<br>
� � � �this moment onwards starts receiving the �live� messages, and is<br>
� � � �pretty much unaware of what�s happened; it simply just receives<br>
� � � �the whole history.<br>
<br>
� � � �Now I am under the impression that there�s a fundamental flaw in<br>
� � � �this reasoning, since the proxy cannot know when the message<br>
� � � �history has completed and he can switch the client to the live<br>
� � � �stream, because while it is replaying historical messages to the<br>
� � � �client other live messages might be arriving on the proxy�s<br>
� � � �queue, thus getting into an endless loop.<br>
<br>
� � � �I�m probably missing something obvious, any advice about how<br>
� � � �others would approach this scenario is appreciated.<br>
<br>
� � � �Simone<br>
<br>
� � � �_________________________________________________________________________________________________________________<br>
� � � �Questo messaggio e da intendersi esclusivamente ad uso del<br>
� � � �destinatario e puo contenere informazioni che sono di natura<br>
� � � �privilegiata, confidenziale<br>
� � � �o non divulgabile secondo le leggi vigenti. Se il lettore del<br>
� � � �presente messaggio non e il destinatario designato, o il<br>
� � � �dipendente/agente responsabile<br>
� � � �per la consegna del messaggio al destinatario designato, si<br>
� � � �informa che ogni disseminazione, distribuzione o copiatura di<br>
� � � �questa comunicazione e<br>
� � � �strettamente proibita anche ai sensi del decreto legislativo<br>
� � � �196/03 . Se avete ricevuto questo messaggio per errore, vi<br>
� � � �preghiamo di notificarcelo<br>
� � � �immediatamente a mezzo e-mail di risposta e successivamente di<br>
� � � �procedere alla cancellazione di questa e-mail e relativi<br>
� � � �allegati dal vostro sistema.<br>
� � � �_________________________________________________________________________________________________________________<br>
� � � �This message is intended only for the use of the addressee and<br>
� � � �may contain information that is privileged, confidential and<br>
� � � �exempt from<br>
� � � �disclosure under applicable law. If the reader of this message<br>
� � � �is not the intended recipient, or the employee or agent<br>
� � � �responsible for delivering the<br>
� � � �message to the intended recipient, you are hereby notified that<br>
� � � �any dissemination, distribution or copying of this communication<br>
� � � �is strictly<br>
� � � �prohibited. If you have received this e-mail in error, please<br>
� � � �notify us immediately by return e-mail and delete this e-mail<br>
� � � �and all attachments from<br>
� � � �your system.<br>
� � � �_________________________________________________________________________________________________________________<br>
<br>
<br>
� � � �_______________________________________________<br>
� � � �rabbitmq-discuss mailing list<br>
� � � �<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br></div></div>
� � � �<mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a>><div><div></div><div class="h5"><br>
� � � �<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
� �_________________________________________________________________________________________________________________<br>
� �Questo messaggio � da intendersi esclusivamente ad uso del<br>
� �destinatario e pu� contenere informazioni che sono di natura<br>
� �privilegiata, confidenziale<br>
� �o non divulgabile secondo le leggi vigenti. Se il lettore del<br>
� �presente messaggio non � il destinatario designato, o il<br>
� �dipendente/agente responsabile<br>
� �per la consegna del messaggio al destinatario designato, si informa<br>
� �che ogni disseminazione, distribuzione o copiatura di questa<br>
� �comunicazione �<br>
� �strettamente proibita anche ai sensi del decreto legislativo 196/03<br>
� �. Se avete ricevuto questo messaggio per errore, vi preghiamo di<br>
� �notificarcelo<br>
� �immediatamente a mezzo e-mail di risposta e successivamente di<br>
� �procedere alla cancellazione di questa e-mail e relativi allegati<br>
� �dal vostro sistema.<br>
� �_________________________________________________________________________________________________________________<br>
� �This message is intended only for the use of the addressee and may<br>
� �contain information that is privileged, confidential and exempt from<br>
� �disclosure under applicable law. If the reader of this message is<br>
� �not the intended recipient, or the employee or agent responsible for<br>
� �delivering the<br>
� �message to the intended recipient, you are hereby notified that any<br>
� �dissemination, distribution or copying of this communication is<br>
� �strictly<br>
� �prohibited. If you have received this e-mail in error, please notify<br>
� �us immediately by return e-mail and delete this e-mail and all<br>
� �attachments from<br>
� �your system.<br>
� �_________________________________________________________________________________________________________________<br>
<br>
<br>
� �_______________________________________________<br>
� �rabbitmq-discuss mailing list<br>
� �<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br></div></div>
� �<mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a>><div><div></div><div class="h5"><br>
� �<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
_________________________________________________________________________________________________________________<br>
Questo messaggio � da intendersi esclusivamente ad uso del destinatario<br>
e pu� contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente<br>
messaggio non � il destinatario designato, o il dipendente/agente<br>
responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che<br>
ogni disseminazione, distribuzione o copiatura di questa comunicazione �<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se<br>
avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere<br>
alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may<br>
contain information that is privileged, confidential and exempt from<br>
disclosure under applicable law. If the reader of this message is not<br>
the intended recipient, or the employee or agent responsible for<br>
delivering the<br>
message to the intended recipient, you are hereby notified that any<br>
dissemination, distribution or copying of this communication is strictly<br>
prohibited. If you have received this e-mail in error, please notify us<br>
immediately by return e-mail and delete this e-mail and all attachments<br>
from<br>
your system.<br>
_________________________________________________________________________________________________________________<br>
<br>
<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote><div><div></div><div class="h5">
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div></div>