[rabbitmq-discuss] question on message delivery

Tue Sep 28 20:23:51 BST 2010

In my Groovy Map/Reduce framework, I've got everything mapping and reducing but I'm having a little trouble making sure everything that needs to know a key change has occurred is notified correctly.

The flow is basically: an SQL statement is executed and individual records are submitted to the Map phase via AMQP message. The map closure calls emit() N number of times, which sends that data to the reduce closure via AMQP message. When a key change occurs, I send a control message, which has its own consumer inside the (Spring AMQP) message handler.

My question is basically: how do I know I've fully processed all potential messages for a given key? If I send a control message out, I'm going to assume there's at least a theoretical possibility of consuming that control message before all map messages have been delivered to the handler. I probably need a two-phase key switch. The handler should be alerted that a key change has occurred by the control message, but the handler itself should decide whether a key change has yet occurred.

Clear as mud? :)

I'm basically wondering if message delivery can occur out-of-order if consumers are bound to different exchanges? Assuming the control message of a key change is published after all the map messages go out (but on a different exchange) can I rely on the map and reduce handlers (which are separate consumers) to watch for key changes in the usual fashion? e.g.

lastKey = null;
...
void onMessage(msg) {
  ...
  if(newKey != lastKey) {
    keyChange();
    lastKey = newKey;
  }
  ...
  invokeAfterClosure(row);
}

Jon Brisbin
Portal Webmaster
NPC International, Inc.