[rabbitmq-discuss] Federation Topologies & Limitations

[Josh West]

Hi Michael,

Wow great response.  Thanks for the details!

Do you think I could accomplish something similar with federated queue's 
and appropriate federation policies?

Thanks.

On 01/15/2014 07:03 PM, Laing, Michael wrote:
> Not seeing any other answers (yet)...
>
> We have experimented with many topologies and have tried federation, 
> but the design that has proven the most rugged and simplest and is in 
> production uses shovels.
>
> We have a core cluster that runs an exchange: the 'postoffice', This 
> cluster has a 'well-known' address so the retail instances can find 
> it. Each retail instance that comes up creates and runs 2 shovels:
>
> 1) the shovel from the core defines a source queue in the core with an 
> instance identifier, e.g. 'outbox_i-12345', and a binding to the core 
> postoffice exchange including that identifier (its address - more 
> about that later).
>
> 2) the shovel to the core uses the instance's 'outbox' queue as 
> source, which is bound to the instance postoffice exchange and 
> specifies the destination exchange as postoffice.
>
> We use 'addresses' as routing and binding keys. Our addresses have 3 
> parts: 'action', 'from_address', 'to_address'. The parts are separated 
> by periods.
>
> Each address has parts: we use 'region', 'product', 'service', 
> 'instance_id', 'pid'. Also separated by periods.
>
> Hence the 'address' of a retail instance is, e.g.:
> action: any action => *
> from_address:
>   region: from any => * (the postoffice actually spans multiple regions)
>   product: from any => * (identifies an internal customer)
>   service: from any => * (an app or a level of service)
>   instance: from any => *
>   pid: from any => *
> to_address:
>   region: mine => us-west-2
>   product: for any => *
>   service: for any => *
>   instance: mine => i-12345
>   pid: for any => *
>
> So that is: '*.*.*.*.*.*.us-west-2.*.*.i-12345.*' meaning the instance 
> will accept anything sent to its region and instance-id.
>
> A typical routing key might be:
> 'sub-resp.eu-west-1.messaging.update.i-98765.444.us-west-2.messaging.gateway.i-12345.-'
> meaning that a service running on a particular pid in another instance 
> is sending our instance a message for a different service, same 
> product, on an unknown pid.
>
> There should be an app queue bound to the postoffice on the instance 
> with an appropriate key to receive the message.
>
> We have standard routines, of course, to handle the creation and 
> manipulation of keys. Plus our deployments are automated - so all 
> those asterisks are quite easy to handle in practice although a little 
> hard to look at!
>
> And of course you can come up with your own addressing scheme to match 
> your requirements that could be simpler.
>
> This is just a straightforward bus implementation, made possible by 
> the flexibility of the RabbitMQ routing/binding scheme. We have found 
> it very useful and have extended the 'postoffice' across all of our 
> instances in multiple regions with good result.
>
> Michael Laing
> NYTimes
>
>
>
> On Wed, Jan 15, 2014 at 2:19 PM, Josh West <jsw at one.com 
> <mailto:jsw at one.com>> wrote:
>
>     Hi folks,
>
>     I'm currently working with the federation plugin in order to
>     connect potentially 100-200 different brokers.  The purpose is to
>     provide a virtual message bus, where systems on any broker can
>     send messages to other systems that are likely connected to a
>     different broker. Basically, a communication network / message bus.
>
>     I am experimenting with different architectures and it seems that
>     in order to prevent message duplications, I'll have to go with a
>     complete graph and max-hops set to 1 -- or in other words, each
>     broker will have to directly federate with the other 100-200 brokers.
>
>     I'm wondering about the limitations (if they exist) of the
>     federation plugin or this style of architecture.  There isn't much
>     information out there on the pro's and con's of different
>     topologies (aside from obvious issues with a large ring topology).
>
>     Can anybody elaborate or add any input?
>
>     Thanks!
>
>     -- 
>     Josh West
>     One.com - http://www.one.com
>
>     _______________________________________________
>     rabbitmq-discuss mailing list
>     rabbitmq-discuss at lists.rabbitmq.com
>     <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>     https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-- 
Josh West
One.com - http://www.one.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140118/1f9e7ac5/attachment.html>