[rabbitmq-discuss] Sequential processing for groups of messages with a multi-consumer setup

Wed Jun 27 10:49:22 BST 2012

Hi Ed,

Here is what i am thinking: 

for option 1, have you tried the RPC style?  take a look at this  
http://www.rabbitmq.com/tutorials/tutorial-six-java.html 

for option 2, i think you may need a smart dispatcher between the queue and 
all those consumers. it will consume messages from the queue and then 
dispatch every single message by ticket number to different consumers. The 
dispatcher need to monitor the status of each consumer, like is it busy or 
which ticket is it processing?

On Tuesday, June 26, 2012 10:30:34 AM UTC+8, Ed Levin wrote:
>
> Hi,
>
> I am considering RabbitMQ for a project with the following requirement and 
> wanted to get some feedback since I am relatively new to message queuing:
>
> To start, I have a queue with a number of consumers assigned for 
> concurrent processing. The messages published to a queue will represent 
> tickets coming from a ticketing system and some may be associated with the 
> same ticket (e.g. same ticket number on different messages in the queue). 
> The problem is that I can only consume (work on) a single message belonging 
> to a given ticket at a time to avoid external concurrency issues. In other 
> words, if consumer A is working on a message representing ticket 1 and, at 
> the same time, consumer B is available and is assigned the next message 
> from the queue that also represents ticket 1, we have a problem.
>
> So far I am considering two possible options:
>
> 1. Dynamically create a temp queue per new unique ticket that arrives, and 
> in effect sort messages into their own queues by ticket number. The problem 
> with this (if even possible) is how to manage consumers for the temp 
> queues. If I assign a single consumer to each queue, the system does not 
> scale, if I assign a consumer to multiple queues I am back to the original 
> problem.
>
> 2.Keep a single queue with multiple consumers assigned and implement a 
> cache to track ticket numbers currently being processed. When a consumer 
> receives a new message, it first checks the cache to see if a given ticket 
> is already being processed. If so, re-queue the message, otherwise add to 
> cache.
>
> I am leaning towards option 2 but appreciate any feedback and wisdom from 
> the community (perhaps I am missing something obvious).
>
> Thank you,
>
> -Ed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120627/39ce95e6/attachment.htm>