<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Putting the list back on CC - I didn't realise I hadn't hit reply-all.<br><div><br><div>Begin forwarded message:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1.0);"><b>From: </b></span><span style="font-family:'Helvetica'; font-size:medium;">"Ryan R." <<a href="mailto:ryan.rajkomar@gmail.com">ryan.rajkomar@gmail.com</a>><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1.0);"><b>Subject: </b></span><span style="font-family:'Helvetica'; font-size:medium;"><b>Re: [rabbitmq-discuss] One Producer, X Consumers where X can change</b><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1.0);"><b>Date: </b></span><span style="font-family:'Helvetica'; font-size:medium;">14 January 2013 10:21:48 GMT<br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1.0);"><b>To: </b></span><span style="font-family:'Helvetica'; font-size:medium;">Tim Watson <<a href="mailto:tim@rabbitmq.com">tim@rabbitmq.com</a>><br></span></div><br><div dir="ltr"><div>Hi,<br><br></div><div>First of all thanks for the quick reply,<br></div><div><br>I'm well aware that trying to use such solution as many issues (mostly the data integrity should a error arise somewhere along the process) and that is indeed problematic.<br>
</div><div>I was actually just looking for a potential way of doing what I need : I have one (or more) webservices whose objects might contains references to objects of another webservice (ideally each WS has its own DB on a different server)<br>
</div><div><br>The issue here is how to keep data integrity when trying to delete the referenced element : if one of the references can't be deleted (for any reason) nothing should be deleted. However here, since data on on different database/webservices/server, I can't do a simple DB transaction.<br>
<br></div><div>Initially I did not really intend to use rabbitMQ for this but only for a notifications system.<br></div><div>But I figured, since having a delay in my deletion process between the request and the actual deletion wasn't an issue, maybe I might be able to find a solution there.<br>
<br></div><div>Finally, and this has nothing to do with my research, for some reason I do not see your post in google groups. How come?<br><br></div><div>Thanks again,<br>Cheers.<br></div></div><div class="gmail_extra"><br>
<br><div class="gmail_quote">2013/1/14 Tim Watson <span dir="ltr"><<a href="mailto:tim@rabbitmq.com" target="_blank">tim@rabbitmq.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hi,<div class="im"><br>
<br>
On 01/14/2013 08:34 AM, Shadowalker wrote:
<blockquote type="cite">Hi again, <br>
Been doing a lot of googling on the queue/topic/listening for
consumed messages count an found this on activemq : <br>
<br>
<a href="http://activemq.apache.org/cms/handling-advisory-messages.html" target="_blank">http://activemq.apache.org/<u></u>cms/handling-advisory-<u></u>messages.html</a><br>
It allows one to check the count of currently listening consumer a
queue/topic. <br>
<br>
</blockquote>
<br></div>
I would not recommend an architecture for distributed resource
tracking based on that. What happens if a consumer is temporarily
disconnected when you go into the check, but reconnects after (or
whilst) the rest of the participants are being updated? You've
introduced even more possibilities for race conditions than before.<br>
<br>
What I would suggest is that you carefully consider whether you
actually need synchronous communications here, as messaging based
architectures inherently de-couple producers from consumers, yet
you've repeatedly attempted to force some 'awareness' of consumers
into the producer whilst discussing this design. I would like to
posit that this reveals an 'impedance mismatch' between your
requirements and the inherently disconnected nature of a queue based
solution. Of course distributed locking is often implemented using
asynchronous communication protocols, but this is usually done at a
*much lower protocol level* - I'd suggest researching Paxos or
similar distributed consensus algorithms to get an idea of what's
involved in designing a reliable solution to this kind of problem.<div class="im"><br>
<br>
<blockquote type="cite">Is there anything like this in rabbit mq ?<br>
</blockquote>
<br></div>
Not that I know of, although it's possible to use the HTTP APIs in
order to track consumers but that is, as I mentioned above, subject
to lots of *nasty* race conditions. You *could* look at using Tony
G's presence exchange (<a href="https://github.com/tonyg/presence-exchange" target="_blank">https://github.com/tonyg/presence-exchange</a>)
to track bindings - although this would complicate your topology
quite a lot, it might make tracking the various participants
plausibly useful, providing you use a known set of binding keys.<div class="im"><br>
<br>
<blockquote type="cite">This might allow me to create a listener that would
only send a message to notify the first manager that the
references were removed.<br>
</blockquote>
<br></div>
I'm not clear on how that helps!? I did have a bit of an early start
this morning though... ;)<div class="im"><br>
<br>
<blockquote type="cite">
Another could be to define the "delete referenrences" message to
live for x consumptions (x being the number of listener on the
"delete references" queue) and add an advisory listener on the
deletion of the message from the queue to process deletion of
initial data.<br>
</blockquote>
<br></div>
That doesn't help at all unless you've actually tracked the number
of acquired messages in the first place. Plus you *can* do that
without 'detecting' the number of consumers. You just insist on
getting a 'make-ref' message from the consumer (with some unique id)
before incrementing the reference count. There's no real difference
between *detecting* the consumer's connection/channel and providing
a ref/lock acquisition queue, except that the latter is probably
more structured, architecturally clearer and quite likely to be more
reliable.<br>
<br>
Even if you used ActiveMQ's detection functionality or RabbitMQ's
management HTTP APIs, the fundamental problem of race conditions
wouldn't go away. Before we go much further discussing various ways
you can design a solution - and I *am* interested in this discussion
BTW - please read
<a href="http://en.wikipedia.org/wiki/Byzantine_fault_tolerance#Byzantine_failures" target="_blank">http://en.wikipedia.org/wiki/Byzantine_fault_tolerance#Byzantine_failures</a>
and make sure you've understood the consequences of nodes *just
disappearing* and then *maybe* coming back later on.<br>
<br>
You've also still not explained what the consequences of loosing
track of resources actually are. If one of your nodes dies, when it
comes back to life has any state been persisted and will that state
thus be used to try and re-acquire or release the 'lock count' for
this resource? What happens if your node sends an 'acquire' request
asynchronously, then starts to write the resource/lock state to its
own local database and dies (e.g., the machine crashes) before
committing the transaction? Because the 'acquire' request was not
synchronous, the master now thinks that your node holds the lock,
whilst the node does *not* think the same. If you bring the node
back online and then start asking for the resource lock, you're
breaking the contract for lock acquisition on that node unless
you're willing to make 'acquire' idempotent, which has its own
pitfalls. If you don't make 'acquire' idempotent, then acquisition
will fail. If you try to handle this by making 'acquire' re-entrant
and then try to release the node's locl, the master will be confused
as it thinks you hold the lock twice and the *lost lock acquisition*
will never be released.<br>
<br>
tldr; this is not a simple problem.<br>
<br>
Cheers,<br>
Tim<br>
<br>
</div>
</blockquote></div><br></div>
</blockquote></div><br></body></html>