[rabbitmq-discuss] RabbitMQ scalability design question

Wed Jul 24 12:57:31 BST 2013

Actually the Consumer that receive the first stop message should receive
also N-1 "I finished my stuff" messages from the other N -1 consumers..
But I think it could be quite easily feasible..summarizing, tell me if this
could be a good workflow:

1) One of the N consumers receive the end-message
2) This consumer opens a new temporary feedback-queue to receive feedbacks
from other consumers and post to a fanout control queue a "wait for end of
processing" to all other consumers (containing also the id of the
feedback-queue)
3) Sender ignores such message, other consumers sends a "I finished my
stuff" message once finished their work to the feedback-queue (whose id is
indicated in the "wait for end of processing" message)
4) Once that the Consumer who initiated the "stop phase" receive all the
N-1 feedbacks (or after a certain timeout) it can delete the temporary
feedback queue and start the map reduce job

What is missing in this scenario is how consumers can be informed of the
number of active consumers (i.e. N)..any help about this? Do you think this
steps could be reasonable?

On Wed, Jul 24, 2013 at 1:10 PM, Michael Klishin <mklishin at gopivotal.com>wrote:

> Flavio Pompermaier:
>
> > If not, could you please give me some more reference or hint about
> implementing the solution you proposed?
>
> just make the consumer that receives the first "stop" message publish a
> message that all
> consumers will get (via a separate queue).
>
> Note that to avoid an infinite loop of messages, you need to mark such
> control messages
> and make sure other consumers don't publish anything, just stop.
>
> Then the consumer that's got the first "stop" message can start a
> map/reduce job.
> --
> MK
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130724/5cfc682d/attachment.htm>