[rabbitmq-discuss] Fwd: Sequential processing for groups of messages with a multi-consumer setup

Tue Jun 26 18:39:12 BST 2012

>> 
>>  ones. A consumer per ticket ID makes sense in 
>> With this kind of design, you would of course have to deal with creating new exchanges and binding them to ticket IDs whenever you do not have an existing binding in place for them already. This will require a control channel, which could take advantage of a default exchange. You might also want to think about housekeeping for ticket IDs that are no longer active (perhaps after some period of time). You'll potentially want to shut down consumers once they've been inactive for some period of time, closing the queue and removing the exchange. The consumer is really in the best position to handle this, as it can easily keep a timer or do non blocking reads.
>> 
>> The advantage of the consistent hash exchange in this architecture, is that it will cope with the changing schematics of exchanges and bindings without requiring explicit reconfiguration all the time, although there may be some scenarios in which explicit application intervention would make sense.
>> 
>> HTH. 
>> 
>> Tim
>> 
>> On 26 Jun 2012, at 03:30, Ed Levin <ed at cloudservicesdepot.com> wrote:
>> 
>>> Hi,
>>> 
>>> I am considering RabbitMQ for a project with the following requirement and wanted to get some feedback since I am relatively new to message queuing:
>>> 
>>> To start, I have a queue with a number of consumers assigned for concurrent processing. The messages published to a queue will represent tickets coming from a ticketing system and some may be associated with the same ticket (e.g. same ticket number on different messages in the queue). The problem is that I can only consume (work on) a single message belonging to a given ticket at a time to avoid external concurrency issues. In other words, if consumer A is working on a message representing ticket 1 and, at the same time, consumer B is available and is assigned the next message from the queue that also represents ticket 1, we have a problem.
>>> 
>>> So far I am considering two possible options:
>>> 
>>> 1. Dynamically create a temp queue per new unique ticket that arrives, and in effect sort messages into their own queues by ticket number. The problem with this (if even possible) is how to manage consumers for the temp queues. If I assign a single consumer to each queue, the system does not scale, if I assign a consumer to multiple queues I am back to the original problem.
>>> 
>>> 2.Keep a single queue with multiple consumers assigned and implement a cache to track ticket numbers currently being processed. When a consumer receives a new message, it first checks the cache to see if a given ticket is already being processed. If so, re-queue the message, otherwise add to cache.
>>> 
>>> I am leaning towards option 2 but appreciate any feedback and wisdom from the community (perhaps I am missing something obvious).
>>> 
>>> Thank you,
>>> 
>>> -Ed
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120626/eca16409/attachment.htm>