<font face="trebuchet ms,sans-serif">I have started implementing it with a layer on top of rabbit, and it gets tricky soon. There are two main challenges:</font><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><br>
</font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">- figure out how to merge the stored and live message streams</font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">- make sure that the replayer component replays messages which partially overlap with those the client subscribed to before asking for stored messages (this is a case of slow message store)</font></div>
<div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><br></font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">To solve the first I have to decorated the messages with a unique identifier so that the client knows when to merge live and stored messages based on id match.</font></div>
<div><font class="Apple-style-span" face="'trebuchet ms', sans-serif">The second one is a bit trickier and I didn't solve it yet. This happens if the component which stores messages is slow and upon replay request is only capable of replaying messages up to a point in time earlier than the first message the client gets from the live stream receives, thus leaving a client with a hole in the message stream.</font></div>
<div><div><br><div class="gmail_quote">On Thu, Jun 2, 2011 at 12:40, Michael Bridgen <span dir="ltr"><<a href="mailto:mikeb@rabbitmq.com">mikeb@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
WRT this:<br>
<br>
/When I was writing it, I thought it would be really easy to, when a new<br>
queue is bound to this exchange, go fetch all the messages in that<br>
bucket and send them to the queue.///<br>
<br>
I’m not sure it is, that was my main question actually. How do you<br>
figure out how to join the messages stored in the bucket and those<br>
arriving live so that the last message you send to the queue from those<br>
picked up from the bucket is followed exactly by its logical successor<br>
from those coming live?<br>
<br>
Say I have messages 1..100 being published to the exchange, each time<br>
one arrives it’s stored. The exchange has received the first 50 messages<br>
and a client performs a binding of a queue to the exchange. So you start<br>
pulling the messages from the storage and sending them to the queue<br>
before forwarding any live messages. While you are pulling them out of<br>
the storage messages 51..60 are published to the exchange, how do you<br>
deal with them? Do you always pull from the storage? When do you stop<br>
pulling from the storage and relay to the queue directly what is being<br>
published to the exchange if you don’t know whether any new messages<br>
have arrived while you were pulling from the storage?<br>
</blockquote>
<br></div>
My inclination with this and similar use cases* is that they deserve their own content class (alongside 'exchange', 'queue', etc.).<br>
<br>
By way of illustration: the last-value-caching exchange, though it was developed as a verification of the exchange type mechanism, is rather awkward; in the main because it is encoded in the existing AMQP operations (bind, consume), and these aren't the right vocabulary for the intended use. See the "Limitations" section of<br>
<a href="https://github.com/squaremo/rabbitmq-lvc-plugin" target="_blank">https://github.com/squaremo/rabbitmq-lvc-plugin</a>.<br>
<br>
There's two natural generalisations of LVC: one is to keep an arbitrary number of values around, and the other is to make the hold higher-order (i.e., instead of replaying values, replay some aggregate of values). Neither of these really fit in AMQP as it stands, not in the available methods or their semantics.<br>
<br>
Anyway. So that this is helpful and not just me kicking AMQP, here is a suggestion: if the consumers are made to co-operate just slightly you can get your proxy idea to work. Just use different queues for the replay and the live. Subscribe the live first and note the most recent message you know about at that time; then publish your saved messages through to that message into the replay queue. The consumer, when it hears back from the proxy, drains the replay queue then starts on the live queue.<br>
<br>
<br>
Michael<br>
<br>
*I call the category "value+updates", and the synchronisation problem you describe above is the essence of it.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
_______________________________________________________________<br>
<br>
*Simone Busoli*<div class="im"><br>
Direzione risorse umane e segreteria generale – Sistemi informativi<br>
gestione sportiva<br>
/Collaboratore/<br>
<br>
*Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>
*T* <a href="tel:%28%2B39%29%200536-949.850" value="+390536949850" target="_blank">(+39) 0536-949.850</a><br></div>
*E* _<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a> <mailto:<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a>>_<br>
_<a href="http://www.ferrari.com" target="_blank">www.ferrari.com</a> <<a href="http://www.ferrari.com/" target="_blank">http://www.ferrari.com/</a>>_<div><div></div><div class="h5"><br>
<br>
*From:* Jon Brisbin [mailto:<a href="mailto:jon@jbrisbin.com" target="_blank">jon@jbrisbin.com</a>]<br>
*Sent:* Wednesday, June 01, 2011 4:59 PM<br>
*To:* Busoli, Simone<br>
*Cc:* <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
*Subject:* Re: [rabbitmq-discuss] How to handle full exchange history<br>
delivery<br>
<br>
Maybe I wasn't being clear:<br>
<br>
I've already written a custom exchange that stores every message it<br>
receives in Riak (<a href="http://wiki.basho.com/" target="_blank">http://wiki.basho.com/</a>). I'm going to be talking about<br>
it next week at Erlang Factory in London, as a matter of fact. :)<br>
<br>
<a href="https://github.com/jbrisbin/riak-exchange" target="_blank">https://github.com/jbrisbin/riak-exchange</a><br>
<br>
When I was writing it, I thought it would be really easy to, when a new<br>
queue is bound to this exchange, go fetch all the messages in that<br>
bucket and send them to the queue.<br>
<br>
Since I'm using Riak, the messages themselves are not all stored in memory.<br>
<br>
Thanks!<br>
<br>
Jon Brisbin<br>
http//<a href="http://jbrisbin.com" target="_blank">jbrisbin.com</a><br>
<br>
------------------------------------------------------------------------<br>
<br>
*From: *"Simone Busoli" <<a href="mailto:Simone.Busoli@ferrari.com" target="_blank">Simone.Busoli@ferrari.com</a>><br>
*To: *<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
*Sent: *Wednesday, June 1, 2011 8:43:58 AM<br>
*Subject: *Re: [rabbitmq-discuss] How to handle full exchange<br>
history delivery<br>
<br>
<br>
That’s what I thought as well, although I’m thinking about a couple<br>
of issues about it:<br>
<br>
· Lack of knowledge about Erlang, which is the language I’d have to<br>
use to build a custom exchange, I guess<br>
<br>
· Amount of memory it would require to store the entire history in<br>
memory. It needn’t be in memory actually, but I’m not sure what<br>
would be the recommended way if the storage has to be carried out by<br>
exchange itself. Mnesia perhaps, would it make sense?<br>
<br>
_______________________________________________________________<br>
<br>
*Simone Busoli*<br>
Direzione risorse umane e segreteria generale – Sistemi informativi<br>
gestione sportiva<br>
/Collaboratore/<br>
<br>
*Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>
*T* <a href="tel:%28%2B39%29%200536-949.850" value="+390536949850" target="_blank">(+39) 0536-949.850</a><br></div></div>
*E* _<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a> <mailto:<a href="mailto:simone.busoli@ferrari.com" target="_blank">simone.busoli@ferrari.com</a>>_<br>
_<a href="http://www.ferrari.com" target="_blank">www.ferrari.com</a> <<a href="http://www.ferrari.com/" target="_blank">http://www.ferrari.com/</a>>_<div><div></div><div class="h5"><br>
<br>
*From:* Jon Brisbin [mailto:<a href="mailto:jon@jbrisbin.com" target="_blank">jon@jbrisbin.com</a>]<br>
*Sent:* Wednesday, June 01, 2011 3:38 PM<br>
*To:* Busoli, Simone<br>
*Cc:* <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
*Subject:* Re: [rabbitmq-discuss] How to handle full exchange<br>
history delivery<br>
<br>
I've been seriously considering adding this functionality to the<br>
Riak-backed custom exchange. It was something I thought I would add<br>
first time through but didn't.<br>
<br>
You'd have to use a custom exchange. Not sure if that's an issue in<br>
your scenario or not.<br>
<br>
<br>
Thanks!<br>
<br>
Jon Brisbin<br>
http//<a href="http://jbrisbin.com" target="_blank">jbrisbin.com</a><br>
<br>
------------------------------------------------------------------------<br>
<br>
*From: *"Simone Busoli" <<a href="mailto:Simone.Busoli@ferrari.com" target="_blank">Simone.Busoli@ferrari.com</a>><br>
*To: *<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
*Sent: *Wednesday, June 1, 2011 5:52:24 AM<br>
*Subject: *[rabbitmq-discuss] How to handle full exchange<br>
history delivery<br>
<br>
Hi,<br>
<br>
I am trying to achieve a scenario in which messages are<br>
published from some data source into a RabbitMQ exchange and<br>
clients should be able to receive the full history of messages<br>
flown through that exchange, regardless of the exact moment in<br>
time when they connect.<br>
<br>
I’ve been thinking to solve it by setting up a proxy which<br>
subscribes to the exchange where messages are published using<br>
its own queue, keeps an internal storage of the messages and<br>
accepts requests from clients.<br>
<br>
When a request from a client arrives it just replays the full<br>
message history to the client by publishing the messages to the<br>
client queue (supplied in the ReplyTo field), *then* binds the<br>
client queue to the messages exchange, so that the client from<br>
this moment onwards starts receiving the “live” messages, and is<br>
pretty much unaware of what’s happened; it simply just receives<br>
the whole history.<br>
<br>
Now I am under the impression that there’s a fundamental flaw in<br>
this reasoning, since the proxy cannot know when the message<br>
history has completed and he can switch the client to the live<br>
stream, because while it is replaying historical messages to the<br>
client other live messages might be arriving on the proxy’s<br>
queue, thus getting into an endless loop.<br>
<br>
I’m probably missing something obvious, any advice about how<br>
others would approach this scenario is appreciated.<br>
<br>
Simone<br>
<br>
_________________________________________________________________________________________________________________<br>
Questo messaggio e da intendersi esclusivamente ad uso del<br>
destinatario e puo contenere informazioni che sono di natura<br>
privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del<br>
presente messaggio non e il destinatario designato, o il<br>
dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si<br>
informa che ogni disseminazione, distribuzione o copiatura di<br>
questa comunicazione e<br>
strettamente proibita anche ai sensi del decreto legislativo<br>
196/03 . Se avete ricevuto questo messaggio per errore, vi<br>
preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di<br>
procedere alla cancellazione di questa e-mail e relativi<br>
allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and<br>
may contain information that is privileged, confidential and<br>
exempt from<br>
disclosure under applicable law. If the reader of this message<br>
is not the intended recipient, or the employee or agent<br>
responsible for delivering the<br>
message to the intended recipient, you are hereby notified that<br>
any dissemination, distribution or copying of this communication<br>
is strictly<br>
prohibited. If you have received this e-mail in error, please<br>
notify us immediately by return e-mail and delete this e-mail<br>
and all attachments from<br>
your system.<br>
_________________________________________________________________________________________________________________<br>
<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br></div></div>
<mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a>><div><div></div><div class="h5"><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
_________________________________________________________________________________________________________________<br>
Questo messaggio è da intendersi esclusivamente ad uso del<br>
destinatario e può contenere informazioni che sono di natura<br>
privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del<br>
presente messaggio non è il destinatario designato, o il<br>
dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si informa<br>
che ogni disseminazione, distribuzione o copiatura di questa<br>
comunicazione è<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03<br>
. Se avete ricevuto questo messaggio per errore, vi preghiamo di<br>
notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di<br>
procedere alla cancellazione di questa e-mail e relativi allegati<br>
dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may<br>
contain information that is privileged, confidential and exempt from<br>
disclosure under applicable law. If the reader of this message is<br>
not the intended recipient, or the employee or agent responsible for<br>
delivering the<br>
message to the intended recipient, you are hereby notified that any<br>
dissemination, distribution or copying of this communication is<br>
strictly<br>
prohibited. If you have received this e-mail in error, please notify<br>
us immediately by return e-mail and delete this e-mail and all<br>
attachments from<br>
your system.<br>
_________________________________________________________________________________________________________________<br>
<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br></div></div>
<mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a>><div><div></div><div class="h5"><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
_________________________________________________________________________________________________________________<br>
Questo messaggio è da intendersi esclusivamente ad uso del destinatario<br>
e può contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente<br>
messaggio non è il destinatario designato, o il dipendente/agente<br>
responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che<br>
ogni disseminazione, distribuzione o copiatura di questa comunicazione è<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se<br>
avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere<br>
alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may<br>
contain information that is privileged, confidential and exempt from<br>
disclosure under applicable law. If the reader of this message is not<br>
the intended recipient, or the employee or agent responsible for<br>
delivering the<br>
message to the intended recipient, you are hereby notified that any<br>
dissemination, distribution or copying of this communication is strictly<br>
prohibited. If you have received this e-mail in error, please notify us<br>
immediately by return e-mail and delete this e-mail and all attachments<br>
from<br>
your system.<br>
_________________________________________________________________________________________________________________<br>
<br>
<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote><div><div></div><div class="h5">
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div></div>