[rabbitmq-discuss] Recommended python library for simple use?

[Allen Fowler]

> > The one-and-only producer, I had thought, would do something like sleep for a 
> few seconds, query the queue size, and if it's getting low, generate a new block 
> of tasks.
> 
> How are you going to do this from the client?
> 

Well...  i don't know.  :)

As i mentioned, I was planning to use python's multiprocessing.Queue, and that certainly does supply the ability to query the current queue size. 

I am trying to figure out if I can to this in RabbitMQ.  


> > Is this not advisable in RabbitMQ based systems?
> 
> Provided you can answer the question above (which incidentally does
> have a solution) then I don't see this being an issue per se. From an
> architectural perspective you might say that it couples producers and
> consumers, which is what a consumer driven message broker tries to
> avoid in general. It could also be argued that it makes multicasting
> more difficult since you would be bypassing the exchange concept. But
> consumers can share queues as well, which might actually what you want
> in your use case.  And, you could always architect in some kind of
> command mechanism that abstracts out the queue control from the
> producers.
> 

As a AMPQ newbie I am not quite sure what you mean.

To clarify:  In the problem at hand, there is a very large list of tasks to complete stored in a proprietary system.   The proprietary system is constantly reshuffling / editing the list of tasks, and waits for an external "execution engine" to ask for some tasks to work on.  These are marked as "in progress", and eventually "done".  

However, "execution engine" should only be asking for <300 tasks to be marked "in progress" at any time, so as to allow the main system to modify as many tasks as it needs to.

I was hoping to use AMQP in the "execution engine" to reliably distribute these tasks to ~20 workers.  After the task is done the "execution engine" needs to tell the main system to mark that task as done.

Perhaps AMPQ is the wrong tool for the job?




> > Basically most tasks only need to be done in a "loose FIFO" manner.   Some 
> tasks will wind up sharing a particular attribute, and so their relative order 
> must be followed.  The pool of possible values for this attribute is unlimited, 
> but for any given task block, about 1/3 will share the attribute value with at 
> least one other task.
> 
> If your tasks were expressible as numeric priorities and Rabbit
> actually implemented priorities, then that might be a option. But
> unfortunately it doesn't yet. If you can get away with a priority
> queue, you could always put one into your consumer and then set a
> larger prefetch window so that the PQ in your app has more than one
> message to prioritize.
> 

Not sure what you mean...  can you point me to the docs on this?

Worst case I will just dedicate ~5 single worker queues to just handle the pure "in order" streams separately.

(Assuming RabbitMQ can even be used here... see above.)


AF