[rabbitmq-discuss] Advice on simplifying design

Thu May 26 22:19:15 BST 2011

I'm working with RabbitMQ and wondering if anyone has some suggestions to
simplify my design or follow more rabbit-specific idioms.

Currently we receive messages in a packaged format, and dump those into a
single queue for unpackaging later (so far so good).

We have a listener on that queue that unpacks these messages, and dumps them
into a queue-per-customer kind of approach. We have ~15,000 potential
customers, so we've approached it as follows:

 - Every so often, do a call to the management API to get a list of queues
that have messages ready
 - iterate over the queues and process any messages waiting there (in a
separate thread pool)
 - delete the queue if it's empty

Since the scheduled management API approach isn't a very realtime operation
(especially with a few thousand queues - it can be very slow) we also have
an approach where we have a 'queue-updated' queue, which we listen on and
when a message is received there, we process the appropriate queue. Since
processing the queue entries can take a long time, we don't tie the
'queue-updated' message success to the actual processing of the consumer
queues, but use them as hints that we should check a particular consumer
queue.

Is there a better/simpler approach? Perhaps a listener on the exchange
itself? The queue-per-customer approach is advantageous for us given how we
interact with the data - we can batch up messages for the same customer and
save a lot of processing and I/O.

Thanks,

 -Dave

-- 
Dave LeBlanc <david.leblanc at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110526/74d0fe49/attachment.htm>