[rabbitmq-discuss] What is the best option for work sharing that needs topic like selective message receival?

[Marek Majkowski]

On Mon, Jan 31, 2011 at 19:39, Enno Shioji <eshioji at gmail.com> wrote:
>> I'm afraid the problem needs some more explanation.
>>  - what's wrong with just one fat queue that is shared between workers?
> Some workers can perform certain categories of jobs, while some
> workers can perform other categories of job. I.e. workers cannot
> perform all category of jobs. They can only perform a subset of job
> category.

Great.
You've just defined your routing methodology.
I guess the same relations should be represented
using bindings and queues.

The simplest thing to do, would be probably to have
on queue per category, and if a consumer can do
jobs from multiple categories - it should bind to multiple
queues.

On the other hand queues are relatively heavyweight,
and it would be more efficient to have only one queue per
consumer. But that's harder to manage dynamically.

> So, if there is a fat queue, many consumers will pull
> messages that they cannot consume and would have to put those back
> into the queue (which I think would be very inefficient).
>
>>  - are all the tasks going to take roughly the same time?
> At the current moment, yes.
>
>>  - do you need to prioritize the tasks?
> It would be nice to roughly prioritize the tasks (it doesn't have to
> be strict), but it'd be more of a "nice to have" requirement...
>
>
>
> Regards,
> Enno
>
>
>
>
>
> On Mon, Jan 31, 2011 at 11:36 PM, Marek Majkowski <majek04 at gmail.com> wrote:
>> On Mon, Jan 31, 2011 at 09:40, Enno Shioji <eshioji at gmail.com> wrote:
>>> I have an app. I would like to migrate to AMQP based architecture. The
>>> main component/functionality of this app. is to distribute work based
>>> on some rules.
>>>
>>> After learning a little bit about AMQP, I concluded that the way to
>>> achieve this is to have the producer send "jobs" to a direct exchange
>>> with routing keys that indicate the nature of the jobs (let's call it
>>> "job category"), and having queues that binds to each unique routing
>>> key, and then having consumers listening on these queues. Multiple
>>> consumers may listen to the same queue. One job should only be
>>> performed once system-wide (but is not strictly required as long as
>>> performance loss is not significant).
>>>
>>> My problem is that the number of such "job category" increases
>>> overtime. E.g. if in the beginning there is only one routing keys such
>>> as "key.stuff.1.A", overtime keys like "key.stuff.2.A" or
>>> "key.stuff.2.B" will start to emerge and eventually such categories
>>> will reach a large number. However, older categories will eventually
>>> become obsolete so at a given time I would have at most something like
>>> 20K different "job categories". But overtime I would have a LOT of job
>>> categories. (Lifetime of each "job category" depends and can be
>>> anything from 1 day to 1 seconds etc..)
>>>
>>> I'm wondering if there is a better way to achieve what I want.
>>>
>>> If this helps, everything can be transient and non durable. Message
>>> loss can be tolerated as long as it's not too frequent (e.g. once per
>>> 10K msgs etc.)
>>
>> I'm afraid the problem needs some more explanation.
>>  - what's wrong with just one fat queue that is shared between workers?
>>  - are all the tasks going to take roughly the same time?
>>  - do you need to prioritize the tasks?
>>
>> Cheers,
>>  Marek
>>
>