[rabbitmq-discuss] RabbitMQ Implementation

[Phill Pafford]

Here is a little about the setup I have currently. ( 
http://stackoverflow.com/questions/14594569/rabbitmq-implementation )

- REST API to push ( POST ) data into a queue
- The Queue has a Consumer that's always running and Produces to en Exchange
- The Exchange routes to several other Queues ( like 20+ )
- Each of the ( 20+ ) Queues does a specific task ( The Consumers always 
runs as well )
- Cron job runs to check if all the ( 20+ ) Tasks are completed and 
Produces to yet another Queue

I'm not sure I like the Consumers running all the time as each Consumer 
utilizes around 300MB of Ram ( I think it's MBs, it's not in front of me at 
the moment ) and I'm looking for another implementation.

        M <-- Message coming from REST API
        |
        |
        +-First Queue
        |
        |
        | <-- The Exchange
       /|\
      / | \
     /  |  \ <-- bind to multiple queues ( 20+ )
    Q1  Q2  Q3 <-- Each Queue is a task that must be completed
    
    
        | <-- CRON runs to check if all queues above have completed
        |
        |
        Q4 <-- Queues 1,2 and 3 must finish first before Queue 4 can start
        |
        C <-- Consumer 

I the related question below it was suggested to use RPC but the problem 
with this is the RPC ( to my understanding ) Will have multiple instances. 
This is a resource intense process as is and I think by adding the RPC 
calls it will just bog down the server and then become unresponsive ( 
Please correct me if I'm wrong ).

Another approach was to use the Aggregator pattern 

 - http://www.eaipatterns.com/Aggregator.html

Which looks exactly what I need but I found the documentation limited. Has 
anyone done this pattern? 

My question is I'm not happy with how it's currently implemented and I'm 
looking for ways to improve the process. I'm looking to either get rid of 
the CRON, Implement a new Pattern and Not have the Consumers run all the 
time.

The process currently also only supports a single instance of each 
consumer. It can have multiple consumers but how we have implemented it we 
only wanted one at the time.

This is implemented in PHP, Symfony2 Framework using the RabbitMQBundle

Related Question:

 - 
http://stackoverflow.com/questions/13861459/rabbitmq-wait-for-multiple-queues-to-finish

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130130/fea82468/attachment.htm>