[rabbitmq-discuss] Work Queues design

Wed Jun 19 10:58:58 BST 2013

On 19 Jun 2013, at 10:41, Flavio Pompermaier wrote:

> Hi to everybody,
> I'm new to RabbitMQ but from what I see it could be the solution for a buffer implementation in one of my project.
> In my scenario there will be a big queue of tasks (i.e. document indexing) that should be served as fast as possible,
> 
> Looking at the tutorial I think that what I need is somehow similar to the Work queues (http://www.rabbitmq.com/tutorials/tutorial-two-python.html) plus fair dispatching.
> However, the aforementioned constraint "served as fast as possible" is not clear how to design a correct size of workers, in a scalable way..
> In every example, a worker is a standalone java process, whereas I would like to manage workers more like a pool and take the first available one..
> 

You can open multiple channels on a single connection (i.e., an application running in the JVM can open many connections and many channels per connection - each channel usually being assigned to a thread). As for "take the first available one", that's exactly what the "fair dispatch" section in the tutorial describes.

> 2) If my poolable solution is correct, how can I set up a pool? How do I decide the best size of the pool?

That's totally application specific, depending on how many messages you're expecting to process and so on. You can `re-size' the pool any time you like by adding more consumers - the prefetch-count will ensure that rabbit doesn't send any one consumer more than the specified number of un-acked messages. 

> 3) What if I discover that my server is overloaded and I want to distribute the load?

Add more consumers, as I said above. 

> How can I avoid to store all the queues message in just one server (if I want persistable queues)?

You can set up a cluster and use HA to mirror (i.e., replicate) the queue contents across multiple nodes. See www.rabbitmq.com/ha.html for details. Of course, with HA you'll store all the queue's messages on all of the (mirrored) nodes. If you were thinking of 'sharding' the data across multiple nodes, then we don't offer a solution for that out of the box, though you could probably cook one up yourself using federation (see www.rabbitmq.com/federation.html).

Cheers,
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130619/506424f8/attachment.htm>