[rabbitmq-discuss] Work Queues design

Wed Jul 10 01:10:19 BST 2013

Hi Flavio,

If you haven't already, you may want to investigate setting up competing 
consumers.

Thanks,
Zach

On Wednesday, June 19, 2013 4:41:14 AM UTC-5, Flavio Pompermaier wrote:
>
> Hi to everybody,
> I'm new to RabbitMQ but from what I see it could be the solution for a 
> buffer implementation in one of my project.
> In my scenario there will be a big queue of tasks (i.e. document indexing) 
> that should be served as fast as possible,
>
> Looking at the tutorial I think that what I need is somehow similar to the 
> Work queues (http://www.rabbitmq.com/tutorials/tutorial-two-python.html) 
> plus fair dispatching.
> However, the aforementioned constraint "served as fast as possible" is not 
> clear how to design a correct size of workers, in a scalable way..
> In every example, a worker is a standalone java process, whereas I would 
> like to manage workers more like a pool and take the first available one..
>
> So, I think that I can summarize my questions as:
>
> 1) What is the best RabbitMQ architecture to solve my problems?
> 2) If my poolable solution is correct, how can I set up a pool? How do I 
> decide the best size of the pool?
> 3) What if I discover that my server is overloaded and I want to 
> distribute the load? How can I avoid to store all the queues message in 
> just one server (if I want persistable queues)?
>
> Best regards,
> Flavio
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130709/d2e479a0/attachment.htm>