[rabbitmq-discuss] Routing via user id

Mon Jun 23 07:39:22 BST 2014

On 23 June 2014 at 10:23:47, Sean Joyce (seanmjoyce at gmail.com) wrote:
> > I have a scenario where one producer publishes a stream of messages  
> to an exchange. The exchange has 10 queues (there is an upper limit  
> to the amount of parallel work that can be done - e.g. Database  
> bottleneck) that it routes messages to. The messages can really  
> go to any queue with one exception: only one message with a given  
> user id can be processed at a time.
>  
> All messages with that user id should be routed to the same queue.  
> If there was no database bottleneck I would create a new queue  
> for each user id but I can really only process ten messages at a  
> time.
> I had thought of partitioning the user ids across ten queues but 
> there is no guarantee the incoming messages would be evenly distributed  
> even with different user ids. Thus I could have queues 1,4 and  
> 8 with a backlog while the rest are idle resources.

Consistent hash exchange [1] and one queue per user (named accordingly) sounds
like a pretty close solution. 

As for the 1 message at a time requirement, while you can use RabbitMQ to imitate
a distributed lock of a kind [2], I'd recommend using ZooKeeper, etcd, or Redis
for this.

1. http://hg.rabbitmq.com/rabbitmq-consistent-hash-exchange/file/rabbitmq_v3_3_3/README.md
2. https://www.rabbitmq.com/blog/2014/02/19/distributed-semaphores-with-rabbitmq/
--  
MK  

Software Engineer, Pivotal/RabbitMQ