[rabbitmq-discuss] RabbitMQ scalability design question

[Flavio Pompermaier]

I don't know why I don't receive your reply anymore..I saw them on
news.gname.org..
However, about the point you answered:

This is a good question. Add to this that consumers can crash or become
unavailable because of
network failures. Coordination over RabbitMQ alone will probably be more
difficult than using
Redis for this purpose (as a "presence service", more or less), e.g. using
its set data structures.
Any consumer would add itself to a set on start and any consumer can read
the set and see
who is online. I'm not very up-to-date with Redis but ideally you'd use its
features to detect
consumers leaving.

I think that here Zookeeper could fit very well in this case but I don't
have idea on how to configure it...
In one of the first emails you mentioned it..could you help me in setting
it up and use it to control
active consumers and query how many are them?

Best,
Flavio

On Wed, Jul 24, 2013 at 1:57 PM, Flavio Pompermaier <pompermaier at okkam.it>wrote:

> Actually the Consumer that receive the first stop message should receive
> also N-1 "I finished my stuff" messages from the other N -1 consumers..
> But I think it could be quite easily feasible..summarizing, tell me if
> this could be a good workflow:
>
> 1) One of the N consumers receive the end-message
> 2) This consumer opens a new temporary feedback-queue to receive feedbacks
> from other consumers and post to a fanout control queue a "wait for end of
> processing" to all other consumers (containing also the id of the
> feedback-queue)
> 3) Sender ignores such message, other consumers sends a "I finished my
> stuff" message once finished their work to the feedback-queue (whose id is
> indicated in the "wait for end of processing" message)
> 4) Once that the Consumer who initiated the "stop phase" receive all the
> N-1 feedbacks (or after a certain timeout) it can delete the temporary
> feedback queue and start the map reduce job
>
> What is missing in this scenario is how consumers can be informed of the
> number of active consumers (i.e. N)..any help about this? Do you think this
> steps could be reasonable?
>
> On Wed, Jul 24, 2013 at 1:10 PM, Michael Klishin <mklishin at gopivotal.com>wrote:
>
>> Flavio Pompermaier:
>>
>> > If not, could you please give me some more reference or hint about
>> implementing the solution you proposed?
>>
>> just make the consumer that receives the first "stop" message publish a
>> message that all
>> consumers will get (via a separate queue).
>>
>> Note that to avoid an infinite loop of messages, you need to mark such
>> control messages
>> and make sure other consumers don't publish anything, just stop.
>>
>> Then the consumer that's got the first "stop" message can start a
>> map/reduce job.
>> --
>> MK
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130724/a1581238/attachment.htm>