[rabbitmq-discuss] producer/consumer and every n-th message

[Torsten Curdt]

>> How bad would be the contention n and m are fairly large?
>> ...or do I just need to have enough broker?
>
> If it's just the *numbers* of producers/consumers that are large, rather
> than the message rates, then you can scale beyond a single broker by adding
> nodes. That does work for coping with increased message rates too to a
> degree, but the single queue will become a bottle neck at some point and
> clustering won't help to overcome that.

That's what I thought. But I guess when that happens one could just
create separate "pools" of producers and consumers and just replicate
the setup. Each would have a single queue and that would be fine.

>> Is there a away to tell the fanout consumer to deliver every message
>> to Q1 but only every 10th message to Q2?
>
> No.
>
>> How else could one do this?
>
> Are you just concerned about the 2nd queue filling up? Then perhaps set a
> message ttl on the queue

I am just not sure even just consuming the messages on a single
consumer (and only processing every n-th) will be fast enough. At some
stage one consumer could be just too slow just consuming. Not sure
when that would happen and if that is really is problem - but I would
assume it could become one.

This is about getting a smaller sample of the messages going through
the producer consumer chain. Setting a ttl would probably do the wrong
thing as it would drop the message not uniformly.

cheers,
Torsten
-- 
http://www.yourdailygeekery.com
http://www.torstencurdt.com