[rabbitmq-discuss] Ruby and AMQP

[Mark]

Can you elaborate on how you set up a pool of workers. Does each worker 
boot up the complete rails environment?

On 1/16/12 8:44 AM, Doug Barth wrote:
> Our application makes use of both RabbitMQ backed workers and Resque
> workers. We've found RabbitMQ to be a good fit for quick to process,
> high volume units of work (eg. sending a text message to a phone).
> Resque has been a better fit for work that takes longer to run and
> generally can be volatile (eg. a backend script to clean up database
> tables). We did a simple no-op test of Resque vs. a custom AMQP backed
> worker. Resque's forking model results in a higher floor for how fast
> a single job can be processed: ~10ms for Resque, vs. 0-1ms for AMQP.
> This makes sense because processing a Resque job involves forking a
> new process and re-establishing connections to any external resources.
>
> For workers, we primarily run pools of workers that boot the Rails
> environment and subscribe to queues to receive work. To manage those
> pools, we use god/monit to start and stop them.
>
> Overall, RabbitMQ has worked extremely well for these high performance
> queues. We have a single box running the RabbitMQ server and have seen
> it processing messages durable, persistent messages at up to 2,000/
> sec. We've also had RabbitMQ have as many as 8.1M messages sitting in
> queues waiting to be processed.
>
> For request/response work loads, we've tried a few options. We played
> with RabbitMQ as the communication mechanism, but found there were was
> a performance penalty (likely our code) over a more typical web
> service based connection. We now mostly use HTTP web service calls
> between systems going through a load balancer. That works well enough,
> but we find ourselves spending a fair amount of time dealing with
> figuring out the best way to marshal data to and from JSON. We also
> need to reconfigure the load balancer to add new capacity.
>
> At some point, we'd like to get back to using a queueing system for
> the communication mechanism. This presents two potential benefits for
> us: being able to use msgpack to serialize data between systems, and
> being able to boot new workers and have them start pulling work
> without any config changes.
>
> --
> Doug Barth
> Engineer
> Signal<http://www.signalhq.com>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss