[rabbitmq-discuss] What is the best option for work sharing that needs topic like selective message receival?

[Enno Shioji]

Hi Marek,

Thanks for the reply!

> I'm afraid the problem needs some more explanation.
>  - what's wrong with just one fat queue that is shared between workers?
Some workers can perform certain categories of jobs, while some
workers can perform other categories of job. I.e. workers cannot
perform all category of jobs. They can only perform a subset of job
category. So, if there is a fat queue, many consumers will pull
messages that they cannot consume and would have to put those back
into the queue (which I think would be very inefficient).

>  - are all the tasks going to take roughly the same time?
At the current moment, yes.

>  - do you need to prioritize the tasks?
It would be nice to roughly prioritize the tasks (it doesn't have to
be strict), but it'd be more of a "nice to have" requirement...



Regards,
Enno





On Mon, Jan 31, 2011 at 11:36 PM, Marek Majkowski <majek04 at gmail.com> wrote:
> On Mon, Jan 31, 2011 at 09:40, Enno Shioji <eshioji at gmail.com> wrote:
>> I have an app. I would like to migrate to AMQP based architecture. The
>> main component/functionality of this app. is to distribute work based
>> on some rules.
>>
>> After learning a little bit about AMQP, I concluded that the way to
>> achieve this is to have the producer send "jobs" to a direct exchange
>> with routing keys that indicate the nature of the jobs (let's call it
>> "job category"), and having queues that binds to each unique routing
>> key, and then having consumers listening on these queues. Multiple
>> consumers may listen to the same queue. One job should only be
>> performed once system-wide (but is not strictly required as long as
>> performance loss is not significant).
>>
>> My problem is that the number of such "job category" increases
>> overtime. E.g. if in the beginning there is only one routing keys such
>> as "key.stuff.1.A", overtime keys like "key.stuff.2.A" or
>> "key.stuff.2.B" will start to emerge and eventually such categories
>> will reach a large number. However, older categories will eventually
>> become obsolete so at a given time I would have at most something like
>> 20K different "job categories". But overtime I would have a LOT of job
>> categories. (Lifetime of each "job category" depends and can be
>> anything from 1 day to 1 seconds etc..)
>>
>> I'm wondering if there is a better way to achieve what I want.
>>
>> If this helps, everything can be transient and non durable. Message
>> loss can be tolerated as long as it's not too frequent (e.g. once per
>> 10K msgs etc.)
>
> I'm afraid the problem needs some more explanation.
>  - what's wrong with just one fat queue that is shared between workers?
>  - are all the tasks going to take roughly the same time?
>  - do you need to prioritize the tasks?
>
> Cheers,
>  Marek
>