[rabbitmq-discuss] last sticky wicket on map/reduce

[Marek Majkowski]

On Thu, Sep 30, 2010 at 21:41, Jon Brisbin
<jon.brisbin at npcinternational.com> wrote:
> I've got a pest of a sticky wicket in my map/reduce implementation that's
> using Groovy for the logic and RabbitMQ for the plumbing. It's frustrating
> because I'm so close.
> The problem I'm having is knowing when I'm finished. Using data like this:
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> 8
> 9
> 10
> END
> The "END" goes through a separate consumer thread because it goes out on a
> fanout exchange (it has to go to all workers), so it comes in out-of-order
> from the other data:
> 1    2    3
>           END
> 4    5    6
> END  END
> 7    8    9
> ...etc...
> I can sort of work around this by keeping track of id changes in my
> consumers using the classic "if this.id != last.id" approach. But the last
> record is a tricky one because there's no key change event to trigger
> sending the response back. Unless I simply wait until a timeout has
> occurred, I'm not sure how I can tell when I've collected all the responses
> I'm going to get.


Interesting problem. If you're using a single queue to handle tasks, you can
never be sure that all the tasks have been done. Well, there always can be
a hanged consumer that got one task and never had done it fully!

The simplest approach would be to keep a timeout. If you haven't received
a message in some time - your worker is ready to quit. But that's wrong -
it won't handle the 'hanged worker' situation.

So before exiting the last 'behaving' worker we should make sure that there
are no other 'misbehaving' workers left. The simplest way to know if we're
the last broker is by doing "queue.declare" and looking at the consumer
count for that particular queue. If it's more than one - we're not the last
consumer.

The trick is how to know if the consumers that are left alone are 'behaving'
or not and how to avoid race condition - many workers quitting in exactly
the same moment.

I'd solve it by sending a message 'QUIT' to the queue once we're sure
that no other work is left in the queue. A worker that receives that
QUIT message may think - okay, there is at least one more 'behaving'
worker, I can't see any work to do, he also can't see any more work,
one of us can happily go home.

That approach works fine until a worker receives it's own message.
If that happens that might mean:
 - we're the only left 'behaving' consumer
 - we have a bad luck and the message always gets randomly dispatched
to ourselves.

In the first case, if we're the only one 'behaving' consumer and if
'queue.declare' doesn't claim any left misbehaving consumers, we may
happily quit.

I hope I answered the right question.

Cheers,
  Marek Majkowski