<div dir="ltr">It is possible. But it is a while since I set up the 'postoffice' using federation.<div><br></div><div>Ultimately I settled on shovels because they are simple, reliable, didn't create structures I couldn't thoroughly understand or control, and didn't have the hop count issues.</div>
<div><br></div><div>We auto-generate our configs, alleviating the big pain associated with shovels.</div><div><br></div><div>That said, this year I will look again into federation for at least some of our requirements.</div>
<div><br></div><div>Michael</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jan 18, 2014 at 3:33 PM, Josh West <span dir="ltr"><<a href="mailto:jsw@one.com" target="_blank">jsw@one.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hi Michael,<br>
<br>
Wow great response. Thanks for the details!<br>
<br>
Do you think I could accomplish something similar with federated
queue's and appropriate federation policies?<br>
<br>
Thanks.<div><div class="h5"><br>
<br>
<div>On 01/15/2014 07:03 PM, Laing, Michael
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Not seeing any other answers (yet)...
<div><br>
</div>
<div>We have experimented with many topologies and have tried
federation, but the design that has proven the most rugged and
simplest and is in production uses shovels.</div>
<div>
<br>
</div>
<div>We have a core cluster that runs an exchange: the
'postoffice', This cluster has a 'well-known' address so the
retail instances can find it. Each retail instance that comes
up creates and runs 2 shovels: </div>
<div><br>
</div>
<div>1) the shovel from the core defines a source queue in the
core with an instance identifier, e.g. 'outbox_i-12345', and a
binding to the core postoffice exchange including that
identifier (its address - more about that later).</div>
<div><br>
</div>
<div>2) the shovel to the core uses the instance's 'outbox'
queue as source, which is bound to the instance postoffice
exchange and specifies the destination exchange as postoffice.</div>
<div><br>
</div>
<div>We use 'addresses' as routing and binding keys. Our
addresses have 3 parts: 'action', 'from_address',
'to_address'. The parts are separated by periods.</div>
<div><br>
</div>
<div>Each address has parts: we use 'region', 'product',
'service', 'instance_id', 'pid'. Also separated by periods.</div>
<div><br>
</div>
<div>Hence the 'address' of a retail instance is, e.g.:</div>
<div>action: any action => *</div>
<div>from_address:</div>
<div> region: from any => * (the postoffice actually spans
multiple regions)</div>
<div> product: from any => * (identifies an internal
customer)</div>
<div> service: from any => * (an app or a level of service)</div>
<div> instance: from any => *</div>
<div> pid: from any => *</div>
<div>to_address:</div>
<div> region: mine => us-west-2</div>
<div> product: for any => *</div>
<div> service: for any => *</div>
<div> instance: mine => i-12345</div>
<div> pid: for any => *</div>
<div><br>
</div>
<div>So that is: '*.*.*.*.*.*.us-west-2.*.*.i-12345.*' meaning
the instance will accept anything sent to its region and
instance-id.</div>
<div><br>
</div>
<div>A typical routing key might be: </div>
<div>'sub-resp.eu-west-1.messaging.update.i-98765.444.us-west-2.messaging.gateway.i-12345.-'</div>
<div>meaning that a service running on a particular pid in
another instance is sending our instance a message for a
different service, same product, on an unknown pid.</div>
<div><br>
</div>
<div>There should be an app queue bound to the postoffice on the
instance with an appropriate key to receive the message.</div>
<div><br>
</div>
<div>We have standard routines, of course, to handle the
creation and manipulation of keys. Plus our deployments are
automated - so all those asterisks are quite easy to handle in
practice although a little hard to look at!</div>
<div><br>
</div>
<div>And of course you can come up with your own addressing
scheme to match your requirements that could be simpler.</div>
<div><br>
</div>
<div>This is just a straightforward bus implementation, made
possible by the flexibility of the RabbitMQ routing/binding
scheme. We have found it very useful and have extended the
'postoffice' across all of our instances in multiple regions
with good result.</div>
<div><br>
</div>
<div>Michael Laing</div>
<div>NYTimes</div>
<div><br>
</div>
<div> </div>
<div> </div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Jan 15, 2014 at 2:19 PM, Josh
West <span dir="ltr"><<a href="mailto:jsw@one.com" target="_blank">jsw@one.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi folks,<br>
<br>
I'm currently working with the federation plugin in order to
connect potentially 100-200 different brokers. The purpose
is to provide a virtual message bus, where systems on any
broker can send messages to other systems that are likely
connected to a different broker. Basically, a communication
network / message bus.<br>
<br>
I am experimenting with different architectures and it seems
that in order to prevent message duplications, I'll have to
go with a complete graph and max-hops set to 1 -- or in
other words, each broker will have to directly federate with
the other 100-200 brokers.<br>
<br>
I'm wondering about the limitations (if they exist) of the
federation plugin or this style of architecture. There
isn't much information out there on the pro's and con's of
different topologies (aside from obvious issues with a large
ring topology).<br>
<br>
Can anybody elaborate or add any input?<br>
<br>
Thanks!<span><font color="#888888"><br>
<br>
-- <br>
Josh West<br>
One.com - <a href="http://www.one.com" target="_blank">http://www.one.com</a><br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</font></span></blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
rabbitmq-discuss mailing list
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Josh West
One.com - <a href="http://www.one.com" target="_blank">http://www.one.com</a></pre>
</div></div></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>