[rabbitmq-discuss] RabbitMQ Implementation

Wed Jan 30 19:34:26 GMT 2013

I've just replied to your question:
http://stackoverflow.com/a/14612183/342013

Regards,

Alvaro

On Wed, Jan 30, 2013 at 4:53 PM, Phill Pafford <phillpafford at gmail.com>wrote:

> Here is a little about the setup I have currently. (
> http://stackoverflow.com/questions/14594569/rabbitmq-implementation )
>
> - REST API to push ( POST ) data into a queue
> - The Queue has a Consumer that's always running and Produces to en
> Exchange
> - The Exchange routes to several other Queues ( like 20+ )
> - Each of the ( 20+ ) Queues does a specific task ( The Consumers always
> runs as well )
> - Cron job runs to check if all the ( 20+ ) Tasks are completed and
> Produces to yet another Queue
>
> I'm not sure I like the Consumers running all the time as each Consumer
> utilizes around 300MB of Ram ( I think it's MBs, it's not in front of me at
> the moment ) and I'm looking for another implementation.
>
>         M <-- Message coming from REST API
>         |
>         |
>         +-First Queue
>         |
>         |
>         | <-- The Exchange
>        /|\
>       / | \
>      /  |  \ <-- bind to multiple queues ( 20+ )
>     Q1  Q2  Q3 <-- Each Queue is a task that must be completed
>
>
>         | <-- CRON runs to check if all queues above have completed
>         |
>         |
>         Q4 <-- Queues 1,2 and 3 must finish first before Queue 4 can start
>         |
>         C <-- Consumer
>
> I the related question below it was suggested to use RPC but the problem
> with this is the RPC ( to my understanding ) Will have multiple instances.
> This is a resource intense process as is and I think by adding the RPC
> calls it will just bog down the server and then become unresponsive (
> Please correct me if I'm wrong ).
>
> Another approach was to use the Aggregator pattern
>
>  - http://www.eaipatterns.com/Aggregator.html
>
> Which looks exactly what I need but I found the documentation limited. Has
> anyone done this pattern?
>
> My question is I'm not happy with how it's currently implemented and I'm
> looking for ways to improve the process. I'm looking to either get rid of
> the CRON, Implement a new Pattern and Not have the Consumers run all the
> time.
>
> The process currently also only supports a single instance of each
> consumer. It can have multiple consumers but how we have implemented it we
> only wanted one at the time.
>
> This is implemented in PHP, Symfony2 Framework using the RabbitMQBundle
>
> Related Question:
>
>  -
> http://stackoverflow.com/questions/13861459/rabbitmq-wait-for-multiple-queues-to-finish
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130130/34dc7aef/attachment.htm>