[rabbitmq-discuss] [rabbitmq-consistent-hash-exchange] Question related to workload distribution

[Emile Joubert]

Hi Josh,

On 24/01/12 23:39, Josh Stone wrote:
> We're looking at using rabbitmq with the consistent hash exchange to
> distribute workload to a set of workers with each worker subscribing
> to one or more topics. It looks like this will work well, until a
> worker along with its topics either comes or leaves, which will
> disrupt the hashing.

Keep in mind that when binding a queue to a consistent hash exchange you
don't bind with a topic. You bind with a number that indicates what
share of messages the queue will receive. Those messages can have any
routing key. If you want workers to handle only particular topics then
you should use a topic exchange.

> Particularly, if a new worker enters and creates some topics on the
> exchange, messages that previously would have been delivered to
> another topic will be directed to the new topic. And if an existing
> worker was operating on some data from a previous message and a new
> message to operate on that data is sent to a new topic, we could have
> a race condition.

For the consistent hash exchange some keys will be unavoidably be
remapped when the bindings change. The advantage is that not all keys
need to be remapped. If your workers can only deal with one or a limited
number of topics then you should not be using a consistent hash exchange.




-Emile