[rabbitmq-discuss] Consistent Hash Exchange

[Matthew Collinge]

Simon,

Thanks for your reply. Here is an example of our use case based on your topology:

X is bound to CHX1 with "MessageTypeA.Something.#" and "MessageTypeB.SomethingElse.#"

P publishes message M1 with routing key: "MessageTypeA.Something.<HashMe>"  - This is delivered to Exchange CHX1
P then publishes another message M2 with a different routing key:  "MessageTypeB.SomethingElse.<HashMe>"  - This is delivered to Exchange CHX1

<HashMe> from message M1 is the same as <HashMe> from message M2 and should be delivered to the same queue say Q1 as they must be consumed by the same consumer.

The way the Consistent Hash Exchange seems to work is that it takes the entire routing key and hashes that. Is there a way to make it use a segment of the routing key?

Cheers

Matt

-----Original Message-----
From: Simon MacMullen [mailto:simon at rabbitmq.com]
Sent: 04 January 2013 17:15
To: Discussions about RabbitMQ
Cc: Matthew Collinge
Subject: Re: [rabbitmq-discuss] Consistent Hash Exchange

So some of this comes down to "it depends what you want to do". If you are doing some routing based on the routing key (before?) the messages hit the CHX then you presumably are using e2e bindings like so:

                 /> Q1
          > CHX1 -> Q2
         /       \> Q3
        /        /> Q4
P -> X --> CHX2 -> Q5
        \        \> Q6
         \       /> Q7
          > CHX3 -> Q8
                 \> Q9

with routing for message type at X and then load balancing at CHX{1,2,3}.

So if the X in question is a topic exchange then option i) should work fine in the scenario above. Putting the hash at either end is fine, the key becomes a tree structure for routing anyway.

I don't know what exchange type you are currently using although I guess topic; if you are using a direct exchange you would either have to switch to topic or change your publishers to publish to one of several CHXs rather than to one DX with one of several routing keys.

Options ii) and iii) struggle with the fact that the CHX doesn't currently support hashing on anything other than the routing key.

Cheers, Simon

On 04/01/13 15:13, matthew.collinge at comparethemarket.com wrote:
> We are looking at using your consistent hash exchange for partitioning
> messages that we need to be processes synchronously based on a
> partition key (GUID in our case). The issue we currently face is that
> we are already using the routing key for subscriptions based on message type.
> As there will be messages of multiple types for the same partition key
> this will result in different routing keys and therefore different
> hashes of those keys for message that we want partitioned into the
> same destination queue.
>
> What would be the best way to go about trying to achieve this? I have
> thought of the following options but do not know the relative merits
> with respect to performance and the inner workings of RabbitMQ:
>
> i)Hash based on the first or last portion of a routing key
> (<HashME>.Iamused.For.Other.Routing)
>
> ii)Hash based on a Message Header
>
> iii)Hash based on a Message Property
>
> Is there any approach that you would advocate or indeed strongly
> advise against?
>
> Thank you in advance.
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


--
Simon MacMullen
RabbitMQ, VMware

________________________________

This message is intended for the stated addressee(s) only and may be confidential. Access to this email by anyone else is unauthorised. Any opinions expressed in this email do not necessarily reflect the opinions of BGL Group Limited and its subsidiaries . Any unauthorised disclosure, use or dissemination, either whole or in part is prohibited. If you are not the intended recipient of this message, please notify the sender immediately.

Please scan all attachments for viruses.