[rabbitmq-discuss] Exchanges, Routing, and AMQP

Wed Nov 12 10:51:53 GMT 2008

Hey Peter,

I'm just keeping the list in the loop.

Ben

---------- Forwarded message ----------
From: Peter Silva <Peter.A.Silva at gmail.com>
Date: Tue, Nov 11, 2008 at 10:12 PM
Subject: Re: [rabbitmq-discuss] Exchanges, Routing, and AMQP
To: Ben Hood <0x6e6562 at gmail.com>

On Tue, Nov 11, 2008 at 6:35 AM, Ben Hood <0x6e6562 at gmail.com> wrote:
>
> Peter,
>
> On Tue, Nov 11, 2008 at 4:42 AM, Peter Silva <Peter.A.Silva at gmail.com> wrote:
>
> This discussion (as an extension to the thread Kirk started) is going
> on too long. If defering the routing type to the bindings is as
> trivial as you maintain, then please enlighten me with a proof of
> concept written in a language of your choice. It would have not have
> to be AMQP specific, just something that shows the basr fundamental
> routing lookup mechanism based on the data structures you are
> proposing.
>
> Code speaks louder than words and I would love to be proven wrong.

Funny you should ask!

I would really appreciate someone who understands switching having a
look at our switch.  We've only used it internally so far, but I
whipped up some .deb's in honour of your request:

http://metpx.sf.net

It's a third generation switch, build from blackbox re-engineering of
the other two generations.  There is developer documentation to get a
loopback configuration working for testing, it's very weather oriented
right now.   We've been working on it for a few years, but the weather
messaging community is, well, kind of scattered, so the installed base
is 1.  It's a very enthusiastic 1, mind you :-)

concept mapping:

AMQP           MetPX
---------            -----------
Exchange      Receiver
Queue           Sender
Binding          Routing

bindings work with two verbs, accept and reject.
We differentiate between regex's and direct mappings for efficiency
reasons (use a table for the direct ones.)
and yeah, we're stuck with a linear search through regex's, which we
try to minimize.

We didn't do routing that matches to exchanges, but almost all the
exchanges use the same routing tables. It implements the rest of the
routing algorithms I have described in python.  Ruthless simplicity
meant that our first database was done using 1 file per message on a
raw file system.  It turns out we can deliver hundreds of messages per
second with that storage and FTP, which was good enough for us.

The packaging is kind of rough, internal people use it a little
differently.  We've been using debian packages for a few months, but
file layout is a little different.

The routing is actually a very small part of the code a few hundred
lines at most.  Most of it is protocol drivers.

The reason for me asking these questions is to understand how to add
AMQP support to metpx.

ready to answer questions, on or offline.

>
>
> > On the other hand,
> > going through long lists of regular expression evaluations can kill
> > performance.
>
> :-)

the key is minimizing the number that have to be searched for each
routing event.   If you treasure speed over flexibility, then just
don't put any regex's in your routing.

Another difference in the design point, is that metpx came from
dealing with existing exchanges, and did not have the luxury of
defining it's own key syntax.   Data producers are often limited in
the names they can supply, and consumers with what they receive
(remember, we deal with files, our keys are essentially file names.)

If we insisted on file naming conventions, we ended up having to
implement separate glue elsewhere to rename things we received into
the file naming convention the switch required.
  It made sense to add regex comprehension to the switch itself to
derive routing keys from the arbitrary file names we would receive.