[rabbitmq-discuss] What is the best option for work sharing that needs topic like selective message receival?

[Enno Shioji]

*forgot to include the mailing list, sorry.

Hi Marek,


Thanks! My worry is though that these categories are very short-lived.
If I create and destroy many queues (e.g. 10-100 per second), would
that be a viable solution in e.g. RabbitMQ?



Regards,
Enno
- Hide quoted text -


On Wed, Feb 2, 2011 at 9:06 PM, Marek Majkowski <majek04 at gmail.com> wrote:
> On Mon, Jan 31, 2011 at 19:39, Enno Shioji <eshioji at gmail.com> wrote:
>>> I'm afraid the problem needs some more explanation.
>>>  - what's wrong with just one fat queue that is shared between workers?
>> Some workers can perform certain categories of jobs, while some
>> workers can perform other categories of job. I.e. workers cannot
>> perform all category of jobs. They can only perform a subset of job
>> category.
>
> Great.
> You've just defined your routing methodology.
> I guess the same relations should be represented
> using bindings and queues.
>
> The simplest thing to do, would be probably to have
> on queue per category, and if a consumer can do
> jobs from multiple categories - it should bind to multiple
> queues.
>
> On the other hand queues are relatively heavyweight,
> and it would be more efficient to have only one queue per
> consumer. But that's harder to manage dynamically.
>
>> So, if there is a fat queue, many consumers will pull
>> messages that they cannot consume and would have to put those back
>> into the queue (which I think would be very inefficient).
>>
>>>  - are all the tasks going to take roughly the same time?
>> At the current moment, yes.
>>
>>>  - do you need to prioritize the tasks?
>> It would be nice to roughly prioritize the tasks (it doesn't have to
>> be strict), but it'd be more of a "nice to have" requirement...
>>
>>
>>
>> Regards,
>> Enno
>>
>>
>>
>>
>>
>> On Mon, Jan 31, 2011 at 11:36 PM, Marek Majkowski <majek04 at gmail.com> wrote:
>>> On Mon, Jan 31, 2011 at 09:40, Enno Shioji <eshioji at gmail.com> wrote:
>>>> I have an app. I would like to migrate to AMQP based architecture. The
>>>> main component/functionality of this app. is to distribute work based
>>>> on some rules.
>>>>
>>>> After learning a little bit about AMQP, I concluded that the way to
>>>> achieve this is to have the producer send "jobs" to a direct exchange
>>>> with routing keys that indicate the nature of the jobs (let's call it
>>>> "job category"), and having queues that binds to each unique routing
>>>> key, and then having consumers listening on these queues. Multiple
>>>> consumers may listen to the same queue. One job should only be
>>>> performed once system-wide (but is not strictly required as long as
>>>> performance loss is not significant).
>>>>
>>>> My problem is that the number of such "job category" increases
>>>> overtime. E.g. if in the beginning there is only one routing keys such
>>>> as "key.stuff.1.A", overtime keys like "key.stuff.2.A" or
>>>> "key.stuff.2.B" will start to emerge and eventually such categories
>>>> will reach a large number. However, older categories will eventually
>>>> become obsolete so at a given time I would have at most something like
>>>> 20K different "job categories". But overtime I would have a LOT of job
>>>> categories. (Lifetime of each "job category" depends and can be
>>>> anything from 1 day to 1 seconds etc..)
>>>>
>>>> I'm wondering if there is a better way to achieve what I want.
>>>>
>>>> If this helps, everything can be transient and non durable. Message
>>>> loss can be tolerated as long as it's not too frequent (e.g. once per
>>>> 10K msgs etc.)
>>>
>>> I'm afraid the problem needs some more explanation.
>>>  - what's wrong with just one fat queue that is shared between workers?
>>>  - are all the tasks going to take roughly the same time?
>>>  - do you need to prioritize the tasks?
>>>
>>> Cheers,
>>>  Marek
>>>
>>
>