<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I've got a pest of a sticky wicket in my map/reduce implementation that's using Groovy for the logic and RabbitMQ for the plumbing. It's frustrating because I'm so close.<div><br></div><div>The problem I'm having is knowing when I'm finished. Using data like this:</div><div><br></div><div>1</div><div>2</div><div>3</div><div>4</div><div>5</div><div>6</div><div>7</div><div>8</div><div>9</div><div>10</div><div>END</div><div><br></div><div>The "END" goes through a separate consumer thread because it goes out on a fanout exchange (it has to go to all workers), so it comes in out-of-order from the other data:</div><div><br></div><div><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">1 2 3</span></font></div><div><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> END</span></font></div><div><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">4 5 6</span></font></div><div><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">END </span></font><span class="Apple-style-span" style="font-family: Courier; font-size: 12px; ">END </span></div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 12px; ">7 8 9</span></div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 12px; ">...etc...</span></div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 12px; "><br></span></div><div><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Georgia; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><div>I can sort of work around this by keeping track of id changes in my consumers using the classic "if this.id != last.id" approach. But the last record is a tricky one because there's no key change event to trigger sending the response back. Unless I simply wait until a timeout has occurred, I'm not sure how I can tell when I've collected all the responses I'm going to get.</div><div><br></div><div>The problem is that I only know how many message I've sent and not how many to expect in return. emit() can be called multiple times from a map phase and the reduce phase can take (records per key) * (emitted) and either rereduce the result or reply back to the requestor. The requestor shouldn't know whether the result has been rereduced or not. It should simply process the return values.</div><div><br></div><div>What am I missing to handle situations like this? Should I introduce another component to this that keeps track of how many messages are sent and received? Maybe put a ZooKeeper install in somewhere and coordinate all this? I've already got Riak integrated, though I'd think ZooKeeper would be better at managing concurrent updates.</div><div><br></div><div>Any help or suggestions here would be greatly appreciated! :)</div><div><br>Jon Brisbin</div><div>Portal Webmaster</div><div>NPC International, Inc.</div><br></div></span><br class="Apple-interchange-newline">
</div>
<br></div></body></html>