[rabbitmq-discuss] RabbitMQ scalability design question

[Michael Klishin]

2013/7/24 Flavio Pompermaier <pompermaier at okkam.it>

> 1) One of the N consumers receive the end-message
> 2) This consumer opens a new temporary feedback-queue to receive feedbacks
> from other consumers and post to a fanout control queue a "wait for end of
> processing" to all other consumers (containing also the id of the
> feedback-queue)
> 3) Sender ignores such message, other consumers sends a "I finished my
> stuff" message once finished their work to the feedback-queue (whose id is
> indicated in the "wait for end of processing" message)
> 4) Once that the Consumer who initiated the "stop phase" receive all the
> N-1 feedbacks (or after a certain timeout) it can delete the temporary
> feedback queue and start the map reduce job
>

This sounds about right.


> What is missing in this scenario is how consumers can be informed of the
> number of active consumers (i.e. N)..any help about this? Do you think this
> steps could be reasonable?
>

This is a good question. Add to this that consumers can crash or become
unavailable because of
network failures. Coordination over RabbitMQ alone will probably be more
difficult than using
Redis for this purpose (as a "presence service", more or less), e.g. using
its set data structures.

Any consumer would add itself to a set on start and any consumer can read
the set and see
who is online. I'm not very up-to-date with Redis but ideally you'd use its
features to detect
consumers leaving.
-- 
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130724/7a6dd666/attachment.htm>