[rabbitmq-discuss] [rabbitmq-consistent-hash-exchange] Question related to workload distribution

[Josh Stone]

Matthew - That is very true about the hash keys.

Regarding workers seeing a message with a routing key they're not
expecting, I'm not sure how a worker would know if a message is
acceptable for them without knowing what the hash ring looks like. If
our workers did mirror the hash algorithm, when a re-mapping occurred
they could check locally to see what work was effected. This brings us
closer to something like Dynamo where each node maintains a view of
the hash ring. In our case, each worker's view of the hash ring would
match that on the consistent hash exchange. This is likely the only
way they could handle key-remappings.

Has anyone gone down this road?

Josh

On Jan 25, 3:33 pm, Matthew Sackman <matt... at rabbitmq.com> wrote:
> On Wed, Jan 25, 2012 at 01:29:35PM -0800, Josh Stone wrote:
> > A followup question. When a new queue is created and a key-remapping
> > occurs, is there any way I could learn which keys were re-mapped? This
> > would allow me to migrate and data/work corresponding to those keys.
> > Perhaps a message could be published to a common queue indicating
> > which keys were re-mapped.
>
> The binding (and thus queue) selected by the consistent hash exachange
> is selected by hashing the message's routing key, and then finding the
> nearest higher binding. Given an infinite number of routing keys will
> hash to the same result, no such message could be constructed to say
> which routing keys have been remapped. The best you could do would be to
> somehow keep track of every routing key the exchange has ever seen, and
> from that figure out which of those would have been routed differently
> in light of the additional-or-loss-of queue/binding. That also isn't
> feasible.
>
> In general, you should probably assume you have relatively little churn
> off such an exchange, and thus if a worker does see a message with a
> routing key it's not expected (caused by eg loss of a queue) then
> propogate the error further up and restart the corresponding session on
> the publishing side so that the entire set of messages with that routing
> key get resent.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss