[rabbitmq-discuss] What is the best option for work sharing that needs topic like selective message receival?

[Enno Shioji]

Hi,


I have an app. I would like to migrate to AMQP based architecture. The
main component/functionality of this app. is to distribute work based
on some rules.

After learning a little bit about AMQP, I concluded that the way to
achieve this is to have the producer send "jobs" to a direct exchange
with routing keys that indicate the nature of the jobs (let's call it
"job category"), and having queues that binds to each unique routing
key, and then having consumers listening on these queues. Multiple
consumers may listen to the same queue. One job should only be
performed once system-wide (but is not strictly required as long as
performance loss is not significant).


My problem is that the number of such "job category" increases
overtime. E.g. if in the beginning there is only one routing keys such
as "key.stuff.1.A", overtime keys like "key.stuff.2.A" or
"key.stuff.2.B" will start to emerge and eventually such categories
will reach a large number. However, older categories will eventually
become obsolete so at a given time I would have at most something like
20K different "job categories". But overtime I would have a LOT of job
categories. (Lifetime of each "job category" depends and can be
anything from 1 day to 1 seconds etc..)

I'm wondering if there is a better way to achieve what I want.

If this helps, everything can be transient and non durable. Message
loss can be tolerated as long as it's not too frequent (e.g. once per
10K msgs etc.)

Thank you very much in advance!




Regards,
Enno