<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I had not really looked at the spring integration stuff for a solution. It looks interesting, though.<div><br></div><div>Thanks for the link...<br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Georgia; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><div><br>Jon Brisbin</div><div>Portal Webmaster</div><div>NPC International, Inc.</div><br></div></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On Sep 30, 2010, at 4:19 PM, Shane Witbeck wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Have you thought about using an Aggregator? Spring Integration offers this: <div><br></div><div><a href="http://static.springsource.org/spring-integration/reference/htmlsingle/spring-integration-reference.html#aggregator">http://static.springsource.org/spring-integration/reference/htmlsingle/spring-integration-reference.html#aggregator</a><div>
<br></div><div>I think Apache Camel offers this too. Both might be overkill in your case but maybe a look at how they're doing it will help.</div><div><br></div><div>HTH,</div><div>Shane</div><div><br></div><div><br><div class="gmail_quote">
On Thu, Sep 30, 2010 at 4:41 PM, Jon Brisbin <span dir="ltr"><<a href="mailto:jon.brisbin@npcinternational.com">jon.brisbin@npcinternational.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word">I've got a pest of a sticky wicket in my map/reduce implementation that's using Groovy for the logic and RabbitMQ for the plumbing. It's frustrating because I'm so close.<div>
<br></div><div>The problem I'm having is knowing when I'm finished. Using data like this:</div><div><br></div><div>1</div><div>2</div><div>3</div><div>4</div><div>5</div><div>6</div><div>7</div><div>8</div><div>9</div>
<div>10</div><div>END</div><div><br></div><div>The "END" goes through a separate consumer thread because it goes out on a fanout exchange (it has to go to all workers), so it comes in out-of-order from the other data:</div>
<div><br></div><div><font face="Courier" size="3"><span style="font-size:12px">1 2 3</span></font></div><div><font face="Courier" size="3"><span style="font-size:12px"> END</span></font></div><div><font face="Courier" size="3"><span style="font-size:12px">4 5 6</span></font></div>
<div><font face="Courier" size="3"><span style="font-size:12px">END </span></font><span style="font-family:Courier;font-size:12px">END </span></div><div><span style="font-family:Courier;font-size:12px">7 8 9</span></div>
<div><span style="font-family:Courier;font-size:12px">...etc...</span></div><div><span style="font-family:Courier;font-size:12px"><br></span></div><div><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div>
<div>I can sort of work around this by keeping track of id changes in my consumers using the classic "if <a href="http://this.id/" target="_blank">this.id</a> != <a href="http://last.id/" target="_blank">last.id</a>" approach. But the last record is a tricky one because there's no key change event to trigger sending the response back. Unless I simply wait until a timeout has occurred, I'm not sure how I can tell when I've collected all the responses I'm going to get.</div>
<div><br></div><div>The problem is that I only know how many message I've sent and not how many to expect in return. emit() can be called multiple times from a map phase and the reduce phase can take (records per key) * (emitted) and either rereduce the result or reply back to the requestor. The requestor shouldn't know whether the result has been rereduced or not. It should simply process the return values.</div>
<div><br></div><div>What am I missing to handle situations like this? Should I introduce another component to this that keeps track of how many messages are sent and received? Maybe put a ZooKeeper install in somewhere and coordinate all this? I've already got Riak integrated, though I'd think ZooKeeper would be better at managing concurrent updates.</div>
<div><br></div><div>Any help or suggestions here would be greatly appreciated! :)</div><div><br>Jon Brisbin</div><div>Portal Webmaster</div><div>NPC International, Inc.</div><br></div></span><br>
</div>
<br></div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>-Shane<br>
</div></div>
</blockquote></div><br></div></body></html>