[rabbitmq-discuss] Advice on simplifying design
david.leblanc at gmail.com
Thu May 26 22:19:15 BST 2011
I'm working with RabbitMQ and wondering if anyone has some suggestions to
simplify my design or follow more rabbit-specific idioms.
Currently we receive messages in a packaged format, and dump those into a
single queue for unpackaging later (so far so good).
We have a listener on that queue that unpacks these messages, and dumps them
into a queue-per-customer kind of approach. We have ~15,000 potential
customers, so we've approached it as follows:
- Every so often, do a call to the management API to get a list of queues
that have messages ready
- iterate over the queues and process any messages waiting there (in a
separate thread pool)
- delete the queue if it's empty
Since the scheduled management API approach isn't a very realtime operation
(especially with a few thousand queues - it can be very slow) we also have
an approach where we have a 'queue-updated' queue, which we listen on and
when a message is received there, we process the appropriate queue. Since
processing the queue entries can take a long time, we don't tie the
'queue-updated' message success to the actual processing of the consumer
queues, but use them as hints that we should check a particular consumer
Is there a better/simpler approach? Perhaps a listener on the exchange
itself? The queue-per-customer approach is advantageous for us given how we
interact with the data - we can batch up messages for the same customer and
save a lot of processing and I/O.
Dave LeBlanc <david.leblanc at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss