[rabbitmq-discuss] Routing key questions/suggestions

[Christian Legnitto]

(meant to send this yesterday but it didn't go through)

On Nov 30, 2010, at 10:06 AM, Matthew Sackman wrote:

> On Tue, Nov 30, 2010 at 09:43:49AM -0800, Christian Legnitto wrote:
>> 1. Why use "." as the routing token separator?
> 
> Well, something has to be chosen, and whatever is chosen is going to be
> inconvenient for someone. I don't know whether there was some reason to
> choose . over anything else.

Yeah, I guess this was inspiration from jms stuff?

> 
>> Can we make it configurable on the broker?
> 
> Well, it's only code. Having had a quick look through, it seems like the
> splitting is currently only done at the use point. So I reckon maybe the
> only change that's needed is in rabbit_exchange_type_topic.erl, change
> the split_topic_key/1 fun to split on whatever you want.

Ok, this might be a way to go. Of course, it would lock me into one implementation and is getting away from the AMQP standard.

> 
>> Can we support sending a magic X-AMQP-ROUTING-SEPERATOR or something
>> header with the message so it can be anything and vary by message?
> 
> Erm that would be substantially more complex.

Yep, but it seems like an interesting way go. That way the publisher can alter the separator based on data. If any routing key tokens have a ".", pick a character that doesn't exist in any of them. When specifying a binding, perhaps use something ugly like foo[:split:]bar[:split:]baz or foo>>bar>>baz or [foo][bar][baz]...or have a way to escape "." and translate any real "." separator into the message specified one. (Note that Matthew had a better suggestion)

I don't know, just throwing cold medicine infused ideas out there ha.

> 
>> 	* It seems like an odd choice to use a character that will show
>> 	up in data a lot
> 
> True. Maybe , would be a better choice given most programming languages
> use , to separate list elements anyway.

For my data this would definitely have been a better choice, though other data sets would likely have a problem. I think it's best to let people choose (I'd actually probably choose one of >~| if it had to be one character). (Note that Matthew had a better suggestion)

> 
>> Example:
>> 
>> I'm listening to commit messages from GitHub. My binding is
>> github.push.*.mycoolproject, which means I want to listen to all push
>> messages by anyone for a project named mycoolproject. If someone's
>> github username has a period in it though, I won't see the messages
>> (github.push.john.smith.mycoolproject). If I bound with
>> github.push.#.mycoolproject I would see the previous message, but I
>> might also get messages for other projects with GitHub users that have
>> mycoolproject in their username
>> (github.push.this.is.mycoolproject.user.project2). I know this isn't
>> the best example, but I'm sick and my head is a bit murky ha.
> 
> Yeah, I know the feeling :/ As I said, _any_ choice is ultimately going
> to collide with something else so you're always going to have to do some
> sort of encoding / escaping yourself. It looks like AMQP doesn't support
> any sort of escaping to prevent the . being interpreted as a semantic .
> so you really are going to have to do translation to something other
> than .
> 
> If, in your case, there is no character that cannot appear in a github
> username (and I'd hope not - surely the whole web supports every unicode
> code point by now...) then you're just not going to be able to solve
> this. This possibly points to AMQP needing to change to support escaping
> of . with a \

Yeah, I figured that was the case. Unfortunately translation ties binding specification to a particular producer/consumer escape mechanism. That is, if my producer escapes "." by replacing it with "[:dot:]", when I bind to "github.push.jim.bob.mycoolproject" I would have to use "github.push.jim[:dot:]bob.mycoolproject". If later I wanted to change this on the producer side (or say I use a 3rd party producer like gitub that doesn't escape the same way) I'd have to modify all the bindings. This is probably an acceptable tradeoff though.

> 
>> 2. Are there are future plans to publish a message with multiple routing keys?
> 
> Not that I'm aware of.
> 
>> 	* I know I can publish the same message multiple times with different routing keys, seems inefficient
> 
> Indeed.
> 
>> For bugzilla messages, when someone changes a bug I send a
>> bug.changed.[field] message. Many tools want to pivot on specific
>> users though. It'd be nice to publish them also as
>> bzuser at example.com.changed.bug so those tools don't have to listen to
>> all changes and group by user themselves, which seems very
>> inefficient.
> 
> You could put _everything_ in the routing key.
> $USER.$BUG_NUMBER.$FIELD
> and then the different tools just need to choose where to put # and *.
> If you prefix each field value with the field name then it ensures that
> such binding keys can anchor in the right place which is useful when the
> range of fields overlap. Eg QA and Assign fields - if you were trying to
> match changes by me to bugs, you'd be ill-advised to just do #.matthew.#
> given that that would match changes _not_ by me but for bugs where I'm
> the QA or assignee. Thus a key of:
> 
> user.matthew.bug.12345.qa.matthew.assignee.matthew.field.status
> 
> then allows a much more precise match of #.qa.matthew.# for example.

Ah, I never really thought of this as I have been trying to treat the routing keys as hierarchical constructs. This feels a bit hacky but will likely work. Do you see other use cases that would support sending multiple routing keys with a given message?

Looks like what I want is a way for consumers to listen to certain key/value pairs in the message. Of course, AMQP isn't set up for that and the broker would need to know about the message formats to do the routing. Using multiple routing keys is pretty much just approximating k/v lookup w/o the broker needing to know about the message payload:

* user.matthew  -> key = user, value = matthew
* bug.12345 -> key = bug, value = 12345
* status.changed -> key = status, value =changed

 I'll play around with your suggestion. Of course, the more data I put in the key the more I come back to my question #1 above :-)

Thanks for the quick response!

Christian