[rabbitmq-discuss] producer/consumer and every n-th message
tcurdt at vafer.org
Fri Jul 8 09:31:34 BST 2011
>> How bad would be the contention n and m are fairly large?
>> ...or do I just need to have enough broker?
> If it's just the *numbers* of producers/consumers that are large, rather
> than the message rates, then you can scale beyond a single broker by adding
> nodes. That does work for coping with increased message rates too to a
> degree, but the single queue will become a bottle neck at some point and
> clustering won't help to overcome that.
That's what I thought. But I guess when that happens one could just
create separate "pools" of producers and consumers and just replicate
the setup. Each would have a single queue and that would be fine.
>> Is there a away to tell the fanout consumer to deliver every message
>> to Q1 but only every 10th message to Q2?
>> How else could one do this?
> Are you just concerned about the 2nd queue filling up? Then perhaps set a
> message ttl on the queue
I am just not sure even just consuming the messages on a single
consumer (and only processing every n-th) will be fast enough. At some
stage one consumer could be just too slow just consuming. Not sure
when that would happen and if that is really is problem - but I would
assume it could become one.
This is about getting a smaller sample of the messages going through
the producer consumer chain. Setting a ttl would probably do the wrong
thing as it would drop the message not uniformly.
More information about the rabbitmq-discuss