[rabbitmq-discuss] What is the best option for work sharing that needs topic like selective message receival?
majek04 at gmail.com
Mon Jan 31 14:36:43 GMT 2011
On Mon, Jan 31, 2011 at 09:40, Enno Shioji <eshioji at gmail.com> wrote:
> I have an app. I would like to migrate to AMQP based architecture. The
> main component/functionality of this app. is to distribute work based
> on some rules.
> After learning a little bit about AMQP, I concluded that the way to
> achieve this is to have the producer send "jobs" to a direct exchange
> with routing keys that indicate the nature of the jobs (let's call it
> "job category"), and having queues that binds to each unique routing
> key, and then having consumers listening on these queues. Multiple
> consumers may listen to the same queue. One job should only be
> performed once system-wide (but is not strictly required as long as
> performance loss is not significant).
> My problem is that the number of such "job category" increases
> overtime. E.g. if in the beginning there is only one routing keys such
> as "key.stuff.1.A", overtime keys like "key.stuff.2.A" or
> "key.stuff.2.B" will start to emerge and eventually such categories
> will reach a large number. However, older categories will eventually
> become obsolete so at a given time I would have at most something like
> 20K different "job categories". But overtime I would have a LOT of job
> categories. (Lifetime of each "job category" depends and can be
> anything from 1 day to 1 seconds etc..)
> I'm wondering if there is a better way to achieve what I want.
> If this helps, everything can be transient and non durable. Message
> loss can be tolerated as long as it's not too frequent (e.g. once per
> 10K msgs etc.)
I'm afraid the problem needs some more explanation.
- what's wrong with just one fat queue that is shared between workers?
- are all the tasks going to take roughly the same time?
- do you need to prioritize the tasks?
More information about the rabbitmq-discuss