[rabbitmq-discuss] Sequential processing for groups of messages with a multi-consumer setup

Wed Jun 27 18:32:08 BST 2012

Rader,

Thank you for the reply. 

I was considering RPC but it is likely to introduce blocking and in this 
scenario it is not important to get a confirmation when a message is 
processed, so async is probably the way to go.

The smart dispatcher idea is interesting, perhaps I can have the dispatcher 
process create/assign temp ticket queues/consumers on the fly to forward 
messages to as needed and that will result in the categorization I am 
looking for.

-Ed

On Wednesday, June 27, 2012 2:49:22 AM UTC-7, Rader wrote:
>
> Hi Ed,
>
> Here is what i am thinking: 
>
> for option 1, have you tried the RPC style?  take a look at this  
> http://www.rabbitmq.com/tutorials/tutorial-six-java.html 
>
> for option 2, i think you may need a smart dispatcher between the queue 
> and all those consumers. it will consume messages from the queue and then 
> dispatch every single message by ticket number to different consumers. The 
> dispatcher need to monitor the status of each consumer, like is it busy or 
> which ticket is it processing?
>
> On Tuesday, June 26, 2012 10:30:34 AM UTC+8, Ed Levin wrote:
>>
>> Hi,
>>
>> I am considering RabbitMQ for a project with the following requirement 
>> and wanted to get some feedback since I am relatively new to message 
>> queuing:
>>
>> To start, I have a queue with a number of consumers assigned for 
>> concurrent processing. The messages published to a queue will represent 
>> tickets coming from a ticketing system and some may be associated with the 
>> same ticket (e.g. same ticket number on different messages in the queue). 
>> The problem is that I can only consume (work on) a single message belonging 
>> to a given ticket at a time to avoid external concurrency issues. In other 
>> words, if consumer A is working on a message representing ticket 1 and, at 
>> the same time, consumer B is available and is assigned the next message 
>> from the queue that also represents ticket 1, we have a problem.
>>
>> So far I am considering two possible options:
>>
>> 1. Dynamically create a temp queue per new unique ticket that arrives, 
>> and in effect sort messages into their own queues by ticket number. The 
>> problem with this (if even possible) is how to manage consumers for the 
>> temp queues. If I assign a single consumer to each queue, the system does 
>> not scale, if I assign a consumer to multiple queues I am back to the 
>> original problem.
>>
>> 2.Keep a single queue with multiple consumers assigned and implement a 
>> cache to track ticket numbers currently being processed. When a consumer 
>> receives a new message, it first checks the cache to see if a given ticket 
>> is already being processed. If so, re-queue the message, otherwise add to 
>> cache.
>>
>> I am leaning towards option 2 but appreciate any feedback and wisdom from 
>> the community (perhaps I am missing something obvious).
>>
>> Thank you,
>>
>> -Ed
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120627/baafccbd/attachment.htm>