[rabbitmq-discuss] Managing consumers (PHP, Symfony2, Capifony)

[Christian Riesen]

Hello everyone

I'm using RabbitMQ over
https://github.com/videlalvaro/RabbitMqBundlewithin the Symfony2 PHP
framework, which in turn uses
https://github.com/videlalvaro/php-amqplib of course.

I have imports to run that do a lot of traffic for about 3 hours per day
and the rest of the day do nothing. RabbitMQ is used to parallelize the
imports (raw data dumped in, processed data is written into storage).

The goal is to have 5 consumers start and in case one dies, automatically
restarted, but always only 5 of them running at the same time. They all run
the same queue and exchange, though that could be different in the future.

I was thinking of using a cronjob that checks every 5 minutes if something
is in the queue (still looking how to see that properly). If there is, it
checks if consumers are running and enough of them, and if not start them
since there are some needed. I'd use the idle timeout feature of the bundle
to shut down the consumers again if there is nothing to do anymore.

We use capifony to deploy our application. Capifony deploys to a folder
like /dir/release/1/ and /dir/release/2/ then sets a symlink from
/dir/current/ to the actual release directory 1 or 2. So the cronjob idea
works well as it would switch to the current deployed version if I point it
to the consumer in the "current" directory. Now I look for a bit a more in
depth way to do this, as I want the workers of the old version stopped when
I deploy. With the cronjob they would start on their own in the right
version of course. The setup runs on a Linux box, so anything I can use on
there would be highly helpful.

To make everything a bit more interesting, I need a sort of locking system.
The queue is one, but multiple producers can fill it up. Then I have to
wait until all the parts in the queue are finished before I can start a
second set of import tasks (that relies on data from the first set). My
current approach was to just monitor the queue and wait for it to empty.
Once the system sees it empty (not 100% sure how to see that yet with the
present tools) it would start the next batch of imports.

This setup I described is not how it has to be, but just my current path of
thinking on how to do it. I'd appreciate alternatives on how to do it. Any
pointers and posts that could help me with one or multiple of these topics
are much appreciated.

Greetings,
Christian Riesen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140310/0a8a5376/attachment.html>