[rabbitmq-discuss] [rabbitmq-consistent-hash-exchange] Question related to workload distribution

Tue Jan 24 23:39:11 GMT 2012

We're looking at using rabbitmq with the consistent hash exchange to
distribute workload to a set of workers with each worker subscribing
to one or more topics. It looks like this will work well, until a
worker along with its topics either comes or leaves, which will
disrupt the hashing.

Particularly, if a new worker enters and creates some topics on the
exchange, messages that previously would have been delivered to
another topic will be directed to the new topic. And if an existing
worker was operating on some data from a previous message and a new
message to operate on that data is sent to a new topic, we could have
a race condition.

I'm wondering how other users are handling this? With other systems
that use consistent hashing such as Dynamo, when the hashing changes
data that would have previously been stored on one node is sent to
another node. When using consistent hashing to distribute workload
though, we can't very easily suspend processing on one worker and send
it to another. Any input on how other users are dealing with this
would be appreciated.

Cheers,
Josh