[rabbitmq-discuss] What is the best option for work sharing that needs topic like selective message receival?

[Marek Majkowski]

On Thu, Feb 3, 2011 at 05:33, Enno Shioji <eshioji at gmail.com> wrote:
> Thanks! My worry is though that these categories are very short-lived.
> If I create and destroy many queues (e.g. 10-100 per second), would
> that be a viable solution in e.g. RabbitMQ?

Yes, but before trusting me with your soul do your own benchmarks
(with the durability, no_wait and other flags set to your requirements)

Marek


> On Wed, Feb 2, 2011 at 9:06 PM, Marek Majkowski <majek04 at gmail.com> wrote:
>> On Mon, Jan 31, 2011 at 19:39, Enno Shioji <eshioji at gmail.com> wrote:
>>>> I'm afraid the problem needs some more explanation.
>>>>  - what's wrong with just one fat queue that is shared between workers?
>>> Some workers can perform certain categories of jobs, while some
>>> workers can perform other categories of job. I.e. workers cannot
>>> perform all category of jobs. They can only perform a subset of job
>>> category.
>>
>> Great.
>> You've just defined your routing methodology.
>> I guess the same relations should be represented
>> using bindings and queues.
>>
>> The simplest thing to do, would be probably to have
>> on queue per category, and if a consumer can do
>> jobs from multiple categories - it should bind to multiple
>> queues.
>>
>> On the other hand queues are relatively heavyweight,
>> and it would be more efficient to have only one queue per
>> consumer. But that's harder to manage dynamically.
>>
>>> So, if there is a fat queue, many consumers will pull
>>> messages that they cannot consume and would have to put those back
>>> into the queue (which I think would be very inefficient).
>>>
>>>>  - are all the tasks going to take roughly the same time?
>>> At the current moment, yes.
>>>
>>>>  - do you need to prioritize the tasks?
>>> It would be nice to roughly prioritize the tasks (it doesn't have to
>>> be strict), but it'd be more of a "nice to have" requirement...
>>>
>>>
>>>
>>> Regards,
>>> Enno
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Jan 31, 2011 at 11:36 PM, Marek Majkowski <majek04 at gmail.com> wrote:
>>>> On Mon, Jan 31, 2011 at 09:40, Enno Shioji <eshioji at gmail.com> wrote:
>>>>> I have an app. I would like to migrate to AMQP based architecture. The
>>>>> main component/functionality of this app. is to distribute work based
>>>>> on some rules.
>>>>>
>>>>> After learning a little bit about AMQP, I concluded that the way to
>>>>> achieve this is to have the producer send "jobs" to a direct exchange
>>>>> with routing keys that indicate the nature of the jobs (let's call it
>>>>> "job category"), and having queues that binds to each unique routing
>>>>> key, and then having consumers listening on these queues. Multiple
>>>>> consumers may listen to the same queue. One job should only be
>>>>> performed once system-wide (but is not strictly required as long as
>>>>> performance loss is not significant).
>>>>>
>>>>> My problem is that the number of such "job category" increases
>>>>> overtime. E.g. if in the beginning there is only one routing keys such
>>>>> as "key.stuff.1.A", overtime keys like "key.stuff.2.A" or
>>>>> "key.stuff.2.B" will start to emerge and eventually such categories
>>>>> will reach a large number. However, older categories will eventually
>>>>> become obsolete so at a given time I would have at most something like
>>>>> 20K different "job categories". But overtime I would have a LOT of job
>>>>> categories. (Lifetime of each "job category" depends and can be
>>>>> anything from 1 day to 1 seconds etc..)
>>>>>
>>>>> I'm wondering if there is a better way to achieve what I want.
>>>>>
>>>>> If this helps, everything can be transient and non durable. Message
>>>>> loss can be tolerated as long as it's not too frequent (e.g. once per
>>>>> 10K msgs etc.)
>>>>
>>>> I'm afraid the problem needs some more explanation.
>>>>  - what's wrong with just one fat queue that is shared between workers?
>>>>  - are all the tasks going to take roughly the same time?
>>>>  - do you need to prioritize the tasks?
>>>>
>>>> Cheers,
>>>>  Marek
>>>>
>>>
>>
>