[rabbitmq-discuss] [rabbitmq-consistent-hash-exchange] Question related to workload distribution

[Josh Stone]

Hi Emile,

Our requirements are to distribute our workload while preventing race
conditions. Using round-robin workload distribution, two workers could
receive concurrent requests to operate on the same data, causing a
race. We can deal with this using distributed locks or consistent
hashing where messages for the same data item are routed to the same
worker. Consistent hashing is ideal, but as we've discussed here we
would need to deal with key-remappings that result from new workers
joining.

Josh

On Jan 25, 4:28 pm, Emile Joubert <em... at rabbitmq.com> wrote:
> Hi Josh,
>
> On 25/01/12 23:47, Josh Stone wrote:
>
> > Regarding workers seeing a message with a routing key they're not
> > expecting, I'm not sure how a worker would know if a message is
> > acceptable for them without knowing what the hash ring looks like. If
> > our workers did mirror the hash algorithm, when a re-mapping occurred
> > they could check locally to see what work was effected. This brings us
> > closer to something like Dynamo where each node maintains a view of
> > the hash ring. In our case, each worker's view of the hash ring would
> > match that on the consistent hash exchange. This is likely the only
> > way they could handle key-remappings.
>
> > Has anyone gone down this road?
>
> I doubt it, because the consistent hash exchange uses a random number
> generator to ensure a fair distribution among all the bindings. What you
> are proposing will require workers to have access to the same random
> number stream and therefore infeasible.
>
> If you share more details about the problem you are trying to solve then
> we may be able to make alternative suggestions. Is there any reason why
> your problem cannot be solved by having multiple workers subscribe to
> the same queue and sharing the load in that way?
>
> -Emile
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss