[rabbitmq-discuss] Managing consumers (PHP, Symfony2, Capifony)

[Alvaro Videla]

Hi,

Others might comment on this, but so far I've seen that people use the
bundle together with supervisord: http://supervisord.org/

About start publishing once a set of tasks has been completed, your
publisher could listen on a queue X, and when one of your workers
consume the last message, it could publish a "done" message back in
the queue X. That way, your producer would know that it can continue
sending the new set of tasks.

About knowing the number of messages in the queue:  you can do that by
sending a passive queue declare, or by using the HTTP management API.

Regards,

Alvaro

On Mon, Mar 10, 2014 at 2:53 PM, Christian Riesen
<christian.riesen at liip.ch> wrote:
> Hello everyone
>
> I'm using RabbitMQ over https://github.com/videlalvaro/RabbitMqBundle within
> the Symfony2 PHP framework, which in turn uses
> https://github.com/videlalvaro/php-amqplib of course.
>
> I have imports to run that do a lot of traffic for about 3 hours per day and
> the rest of the day do nothing. RabbitMQ is used to parallelize the imports
> (raw data dumped in, processed data is written into storage).
>
> The goal is to have 5 consumers start and in case one dies, automatically
> restarted, but always only 5 of them running at the same time. They all run
> the same queue and exchange, though that could be different in the future.
>
> I was thinking of using a cronjob that checks every 5 minutes if something
> is in the queue (still looking how to see that properly). If there is, it
> checks if consumers are running and enough of them, and if not start them
> since there are some needed. I'd use the idle timeout feature of the bundle
> to shut down the consumers again if there is nothing to do anymore.
>
> We use capifony to deploy our application. Capifony deploys to a folder like
> /dir/release/1/ and /dir/release/2/ then sets a symlink from /dir/current/
> to the actual release directory 1 or 2. So the cronjob idea works well as it
> would switch to the current deployed version if I point it to the consumer
> in the "current" directory. Now I look for a bit a more in depth way to do
> this, as I want the workers of the old version stopped when I deploy. With
> the cronjob they would start on their own in the right version of course.
> The setup runs on a Linux box, so anything I can use on there would be
> highly helpful.
>
> To make everything a bit more interesting, I need a sort of locking system.
> The queue is one, but multiple producers can fill it up. Then I have to wait
> until all the parts in the queue are finished before I can start a second
> set of import tasks (that relies on data from the first set). My current
> approach was to just monitor the queue and wait for it to empty. Once the
> system sees it empty (not 100% sure how to see that yet with the present
> tools) it would start the next batch of imports.
>
> This setup I described is not how it has to be, but just my current path of
> thinking on how to do it. I'd appreciate alternatives on how to do it. Any
> pointers and posts that could help me with one or multiple of these topics
> are much appreciated.
>
> Greetings,
> Christian Riesen
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>