[rabbitmq-discuss] Sequential processing for groups of messages with a multi-consumer setup

Tue Jun 26 03:30:34 BST 2012

Hi,

I am considering RabbitMQ for a project with the following requirement and 
wanted to get some feedback since I am relatively new to message queuing:

To start, I have a queue with a number of consumers assigned for concurrent 
processing. The messages published to a queue will represent tickets coming 
from a ticketing system and some may be associated with the same ticket 
(e.g. same ticket number on different messages in the queue). The problem 
is that I can only consume (work on) a single message belonging to a given 
ticket at a time to avoid external concurrency issues. In other words, if 
consumer A is working on a message representing ticket 1 and, at the same 
time, consumer B is available and is assigned the next message from the 
queue that also represents ticket 1, we have a problem.

So far I am considering two possible options:

1. Dynamically create a temp queue per new unique ticket that arrives, and 
in effect sort messages into their own queues by ticket number. The problem 
with this (if even possible) is how to manage consumers for the temp 
queues. If I assign a single consumer to each queue, the system does not 
scale, if I assign a consumer to multiple queues I am back to the original 
problem.

2.Keep a single queue with multiple consumers assigned and implement a 
cache to track ticket numbers currently being processed. When a consumer 
receives a new message, it first checks the cache to see if a given ticket 
is already being processed. If so, re-queue the message, otherwise add to 
cache.

I am leaning towards option 2 but appreciate any feedback and wisdom from 
the community (perhaps I am missing something obvious).

Thank you,

-Ed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120625/60d04659/attachment.htm>