<div dir="ltr"><br>I don't know why I don't receive your reply anymore..I saw them on <a href="http://news.gname.org">news.gname.org</a>..<div>However, about the point you answered:<br><br>This is a good question. Add to this that consumers can crash or become unavailable because of <div>
network failures. Coordination over RabbitMQ alone will probably be more difficult than using<br>Redis for this purpose (as a "presence service", more or less), e.g. using its set data structures.<br>Any consumer would add itself to a set on start and any consumer can read the set and see<br>
who is online. I'm not very up-to-date with Redis but ideally you'd use its features to detect<br>consumers leaving.</div><div><br></div><div>I think that here Zookeeper could fit very well in this case but I don't have idea on how to configure it...</div>
<div>In one of the first emails you mentioned it..could you help me in setting it up and use it to control</div><div>active consumers and query how many are them?</div><div><br></div><div>Best,</div><div>Flavio<br><div class="gmail_extra">
<br><div class="gmail_quote">On Wed, Jul 24, 2013 at 1:57 PM, Flavio Pompermaier <span dir="ltr"><<a href="mailto:pompermaier@okkam.it" target="_blank">pompermaier@okkam.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Actually the Consumer that receive the first stop message should receive also N-1 "I finished my stuff" messages from the other N -1 consumers..<div>But I think it could be quite easily feasible..summarizing, tell me if this could be a good workflow:</div>
<div><br></div><div>1) One of the N consumers receive the end-message</div><div>2) This consumer opens a new temporary feedback-queue to receive feedbacks from other consumers and post to a fanout control queue a "wait for end of processing" to all other consumers (containing also the id of the feedback-queue)</div>
<div>3) Sender ignores such message, other consumers sends a "I finished my stuff" message once finished their work to the feedback-queue (whose id is indicated in the "wait for end of processing" message)</div>
<div>4) Once that the Consumer who initiated the "stop phase" receive all the N-1 feedbacks (or after a certain timeout) it can delete the temporary feedback queue and start the map reduce job</div><div><br></div>
<div>What is missing in this scenario is how consumers can be informed of the number of active consumers (i.e. N)..any help about this? Do you think this steps could be reasonable?</div><div><div class="h5"><div><div class="gmail_extra">
<br><div class="gmail_quote">
On Wed, Jul 24, 2013 at 1:10 PM, Michael Klishin <span dir="ltr"><<a href="mailto:mklishin@gopivotal.com" target="_blank">mklishin@gopivotal.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Flavio Pompermaier:<br>
<div><br>
> If not, could you please give me some more reference or hint about implementing the solution you proposed?<br>
<br>
</div>just make the consumer that receives the first "stop" message publish a message that all<br>
consumers will get (via a separate queue).<br>
<br>
Note that to avoid an infinite loop of messages, you need to mark such control messages<br>
and make sure other consumers don't publish anything, just stop.<br>
<br>
Then the consumer that's got the first "stop" message can start a map/reduce job.<br>
--<br>
MK</blockquote></div></div></div></div></div></div></blockquote></div><div dir="ltr"><p></p><p></p><p></p><p></p></div>
</div></div></div></div>