<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I'm also wondering if anyone uses counts to determine when a job is finished or not. By that I mean, increment a counter for every outgoing message and decrement the counter when a response is received. In the case of a map/reduce job, I'd need to do something like:<div><br></div><div>SQL -> Map phase = +1 (per row)</div><div>Map phase -> Reduce phase = -1 (that we got the original msg) +1 * (num of emit's)</div><div>Reduce phase -> Response|ReReduce = -1 (for emit's) +1 (for response/rereduce)</div><div>[ReReduce -> Response] = -1 +1 (for sending response)</div><div>Response = -1</div><div><br></div><div>Essentially, each step would decrement a counter for the incoming message and increment the counter for the outgoing message. A reduce phase might decrement the counter 1000 times and increment it once. But since the map phase incremented it 1000 times prior, the count after map/reduce would be "1". The response listener would then decrement the counter when it processed the response, see that it's now zero, and know to continue.</div><div><br></div><div>If my goal is to beat processing times on the AS/400 when doing large financial calculations (daily acct'g reports take several hours to generate), I can't really depend on timeouts to make sure I've gathered all my results. I want the job to return as soon as results are ready. I'd like to go to management and show them a 2 hr -> 15 min improvement by using parallel processing.</div><div><br></div><div>I'm just wondering if using ZooKeeper or similar to do distributed, synchronized counters will have enough atomicity to not miss a count incr/decr. If I miss even one, I'm screwed because it'll never get back to zero (or get there prematurely).</div><div><br></div><div>I need a sentence with a question mark or this will definitely go unanswered: are message counters like this a good way to monitor asynchronous, distributed processing state?</div><div><br></div><div>Thanks! :)<br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Georgia; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><div><br>Jon Brisbin</div><div>Portal Webmaster</div><div>NPC International, Inc.</div><br></div></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On Oct 1, 2010, at 8:11 AM, Jon Brisbin wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I had not really looked at the spring integration stuff for a solution. It looks interesting, though.<div><br></div><div>Thanks for the link...<br><div>
<span class="Apple-style-span" style="border-collapse: separate; font-family: Georgia; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><div><br>Jon Brisbin</div><div>Portal Webmaster</div><div>NPC International, Inc.</div><br></div></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On Sep 30, 2010, at 4:19 PM, Shane Witbeck wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Have you thought about using an Aggregator? Spring Integration offers this: <div><br></div><div><a href="http://static.springsource.org/spring-integration/reference/htmlsingle/spring-integration-reference.html#aggregator">http://static.springsource.org/spring-integration/reference/htmlsingle/spring-integration-reference.html#aggregator</a><div>
<br></div><div>I think Apache Camel offers this too. Both might be overkill in your case but maybe a look at how they're doing it will help.</div><div><br></div><div>HTH,</div><div>Shane</div><div><br></div><div><br><div class="gmail_quote">
On Thu, Sep 30, 2010 at 4:41 PM, Jon Brisbin <span dir="ltr"><<a href="mailto:jon.brisbin@npcinternational.com">jon.brisbin@npcinternational.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word">I've got a pest of a sticky wicket in my map/reduce implementation that's using Groovy for the logic and RabbitMQ for the plumbing. It's frustrating because I'm so close.<div>
<br></div><div>The problem I'm having is knowing when I'm finished. Using data like this:</div><div><br></div><div>1</div><div>2</div><div>3</div><div>4</div><div>5</div><div>6</div><div>7</div><div>8</div><div>9</div>
<div>10</div><div>END</div><div><br></div><div>The "END" goes through a separate consumer thread because it goes out on a fanout exchange (it has to go to all workers), so it comes in out-of-order from the other data:</div>
<div><br></div><div><font face="Courier" size="3"><span style="font-size:12px">1 2 3</span></font></div><div><font face="Courier" size="3"><span style="font-size:12px"> END</span></font></div><div><font face="Courier" size="3"><span style="font-size:12px">4 5 6</span></font></div>
<div><font face="Courier" size="3"><span style="font-size:12px">END </span></font><span style="font-family:Courier;font-size:12px">END </span></div><div><span style="font-family:Courier;font-size:12px">7 8 9</span></div>
<div><span style="font-family:Courier;font-size:12px">...etc...</span></div><div><span style="font-family:Courier;font-size:12px"><br></span></div><div><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div>
<div>I can sort of work around this by keeping track of id changes in my consumers using the classic "if <a href="http://this.id/" target="_blank">this.id</a> != <a href="http://last.id/" target="_blank">last.id</a>" approach. But the last record is a tricky one because there's no key change event to trigger sending the response back. Unless I simply wait until a timeout has occurred, I'm not sure how I can tell when I've collected all the responses I'm going to get.</div>
<div><br></div><div>The problem is that I only know how many message I've sent and not how many to expect in return. emit() can be called multiple times from a map phase and the reduce phase can take (records per key) * (emitted) and either rereduce the result or reply back to the requestor. The requestor shouldn't know whether the result has been rereduced or not. It should simply process the return values.</div>
<div><br></div><div>What am I missing to handle situations like this? Should I introduce another component to this that keeps track of how many messages are sent and received? Maybe put a ZooKeeper install in somewhere and coordinate all this? I've already got Riak integrated, though I'd think ZooKeeper would be better at managing concurrent updates.</div>
<div><br></div><div>Any help or suggestions here would be greatly appreciated! :)</div><div><br>Jon Brisbin</div><div>Portal Webmaster</div><div>NPC International, Inc.</div><br></div></span><br>
</div>
<br></div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>-Shane<br>
</div></div>
</blockquote></div><br></div></div>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br></blockquote></div><br></div></body></html>