[rabbitmq-discuss] Consumer management

[Bill Moseley]

How do you manage your consumer/worker processes?  Does anyone look at the
queue lengths and automatically adjust the number of consumers?

Is anyone using http://supervisord.org/ for managing consumer processes?



I have a pool of servers to run a small collection of consumers that do
different tasks.  As load dictates (by looking at messages waiting in
queues) I'd like to be able to dynamically re-balance the mixture of
consumers.  For example, if a lot of requests come in for report generation
and I have a lot of idle consumers that do image manipulation I'd like to
reduce the number of image processing consumers and add more report
generation consumers.

(Many of my consumers are pretty simple in that they just run some command
line tool.)

What's your approach?


And do you tend to write consumers that are small stand-alone programs
(something easy for Supervisor to manage) or write larger (perhaps more
complex) stand-alone apps that listen to a queue and fork (or threads)
workers do handle incoming messages?



-- 
Bill Moseley
moseley at hank.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110131/db248cb6/attachment.htm>