[rabbitmq-discuss] What is the best option for work sharing that needs topic like selective message receival?

[Marek Majkowski]

On Mon, Jan 31, 2011 at 09:40, Enno Shioji <eshioji at gmail.com> wrote:
> I have an app. I would like to migrate to AMQP based architecture. The
> main component/functionality of this app. is to distribute work based
> on some rules.
>
> After learning a little bit about AMQP, I concluded that the way to
> achieve this is to have the producer send "jobs" to a direct exchange
> with routing keys that indicate the nature of the jobs (let's call it
> "job category"), and having queues that binds to each unique routing
> key, and then having consumers listening on these queues. Multiple
> consumers may listen to the same queue. One job should only be
> performed once system-wide (but is not strictly required as long as
> performance loss is not significant).
>
> My problem is that the number of such "job category" increases
> overtime. E.g. if in the beginning there is only one routing keys such
> as "key.stuff.1.A", overtime keys like "key.stuff.2.A" or
> "key.stuff.2.B" will start to emerge and eventually such categories
> will reach a large number. However, older categories will eventually
> become obsolete so at a given time I would have at most something like
> 20K different "job categories". But overtime I would have a LOT of job
> categories. (Lifetime of each "job category" depends and can be
> anything from 1 day to 1 seconds etc..)
>
> I'm wondering if there is a better way to achieve what I want.
>
> If this helps, everything can be transient and non durable. Message
> loss can be tolerated as long as it's not too frequent (e.g. once per
> 10K msgs etc.)

I'm afraid the problem needs some more explanation.
 - what's wrong with just one fat queue that is shared between workers?
 - are all the tasks going to take roughly the same time?
 - do you need to prioritize the tasks?

Cheers,
  Marek