[rabbitmq-discuss] Federation Topologies & Limitations

[Laing, Michael]

It is possible. But it is a while since I set up the 'postoffice' using
federation.

Ultimately I settled on shovels because they are simple, reliable, didn't
create structures I couldn't thoroughly understand or control, and didn't
have the hop count issues.

We auto-generate our configs, alleviating the big pain associated with
shovels.

That said, this year I will look again into federation for at least some of
our requirements.

Michael


On Sat, Jan 18, 2014 at 3:33 PM, Josh West <jsw at one.com> wrote:

>  Hi Michael,
>
> Wow great response.  Thanks for the details!
>
> Do you think I could accomplish something similar with federated queue's
> and appropriate federation policies?
>
> Thanks.
>
>
> On 01/15/2014 07:03 PM, Laing, Michael wrote:
>
> Not seeing any other answers (yet)...
>
>  We have experimented with many topologies and have tried federation, but
> the design that has proven the most rugged and simplest and is in
> production uses shovels.
>
>  We have a core cluster that runs an exchange: the 'postoffice', This
> cluster has a 'well-known' address so the retail instances can find it.
> Each retail instance that comes up creates and runs 2 shovels:
>
>  1) the shovel from the core defines a source queue in the core with an
> instance identifier, e.g. 'outbox_i-12345', and a binding to the core
> postoffice exchange including that identifier (its address - more about
> that later).
>
>  2) the shovel to the core uses the instance's 'outbox' queue as source,
> which is bound to the instance postoffice exchange and specifies the
> destination exchange as postoffice.
>
>  We use 'addresses' as routing and binding keys. Our addresses have 3
> parts: 'action', 'from_address', 'to_address'. The parts are separated by
> periods.
>
>  Each address has parts: we use 'region', 'product', 'service',
> 'instance_id', 'pid'. Also separated by periods.
>
>  Hence the 'address' of a retail instance is, e.g.:
> action: any action => *
> from_address:
>   region: from any => * (the postoffice actually spans multiple regions)
>   product: from any => * (identifies an internal customer)
>   service: from any => * (an app or a level of service)
>   instance: from any => *
>   pid: from any => *
> to_address:
>   region: mine => us-west-2
>   product: for any => *
>   service: for any => *
>   instance: mine => i-12345
>   pid: for any => *
>
>  So that is: '*.*.*.*.*.*.us-west-2.*.*.i-12345.*' meaning the instance
> will accept anything sent to its region and instance-id.
>
>  A typical routing key might be:
>
> 'sub-resp.eu-west-1.messaging.update.i-98765.444.us-west-2.messaging.gateway.i-12345.-'
> meaning that a service running on a particular pid in another instance is
> sending our instance a message for a different service, same product, on an
> unknown pid.
>
>  There should be an app queue bound to the postoffice on the instance
> with an appropriate key to receive the message.
>
>  We have standard routines, of course, to handle the creation and
> manipulation of keys. Plus our deployments are automated - so all those
> asterisks are quite easy to handle in practice although a little hard to
> look at!
>
>  And of course you can come up with your own addressing scheme to match
> your requirements that could be simpler.
>
>  This is just a straightforward bus implementation, made possible by the
> flexibility of the RabbitMQ routing/binding scheme. We have found it very
> useful and have extended the 'postoffice' across all of our instances in
> multiple regions with good result.
>
>  Michael Laing
> NYTimes
>
>
>
>
>
> On Wed, Jan 15, 2014 at 2:19 PM, Josh West <jsw at one.com> wrote:
>
>> Hi folks,
>>
>> I'm currently working with the federation plugin in order to connect
>> potentially 100-200 different brokers.  The purpose is to provide a virtual
>> message bus, where systems on any broker can send messages to other systems
>> that are likely connected to a different broker. Basically, a communication
>> network / message bus.
>>
>> I am experimenting with different architectures and it seems that in
>> order to prevent message duplications, I'll have to go with a complete
>> graph and max-hops set to 1 -- or in other words, each broker will have to
>> directly federate with the other 100-200 brokers.
>>
>> I'm wondering about the limitations (if they exist) of the federation
>> plugin or this style of architecture.  There isn't much information out
>> there on the pro's and con's of different topologies (aside from obvious
>> issues with a large ring topology).
>>
>> Can anybody elaborate or add any input?
>>
>> Thanks!
>>
>> --
>> Josh West
>> One.com - http://www.one.com
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing listrabbitmq-discuss at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> --
> Josh West
> One.com - http://www.one.com
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140118/8a5f9ccc/attachment.html>