<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; "><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">I have been experimenting with federation and thought I would report on the experience as input to development of the capability.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">My goal was to create a 'global exchange' that could reside in any or all of the virtual hosts in my rabbitmq clusters. The purpose of the global exchanges is to provide a 'control bus'-like capability wherein messages published to the exchange in any vhost in any cluster appear in the corresponding exchange in every vhost in every cluster. The messages will be relatively small and message rates relatively low, under 100 per second – probably well under.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">To implement this I created a 'reflector' vhost in each cluster. Within the cluster, I use federation to create bidirectional links in a star configuration between 'reflector' and each other participating vhost in the cluster. We segregate products/projects etc into their own vhosts so there can be many of them in each cluster.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">The clusters in turn are fully connected (link from each cluster to every other cluster) in a federated network using their 'reflector' vhosts.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">For simplicity I use a hop count of 3 for each link. Duplicate messages are ignored.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">Here are 2 message paths of the many that occur, one within the cluster and one between clusters:</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">Message -> exchange:Global/vhost:A/cluster:X </span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">(hop 1) -> </span><span class="Apple-style-span" style="font-family: Courier; ">exchange:Global/vhost:Reflector/cluster:X </span></div><div style="font-family: Calibri, sans-serif; "><span class="Apple-style-span" style="font-family: Courier; ">(hop 2) -> </span><span class="Apple-style-span" style="font-family: Courier; ">exchange:Global/vhost:Reflector/cluster:Y </span></div><div style="font-family: Calibri, sans-serif; "><span class="Apple-style-span" style="font-family: Courier; ">(hop 3) -> </span><span class="Apple-style-span" style="font-family: Courier; ">exchange:Global/vhost:B/cluster:Y</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">Message -> exchange:Global/vhost:A/cluster:X </span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">(hop 1) -> </span><span class="Apple-style-span" style="font-family: Courier; ">exchange:Global/vhost:Reflector/cluster:X </span></div><div style="font-family: Calibri, sans-serif; "><span class="Apple-style-span" style="font-family: Courier; ">(hop 2) -> </span><span class="Apple-style-span" style="font-family: Courier; ">exchange:Global/vhost:C/cluster:X</span></div><div><div style="font-family: Calibri, sans-serif; "><span class="Apple-style-span" style="font-family: Courier; ">(hop 3) -> </span><span class="Apple-style-span" style="font-family: Courier; ">exchange:Global/vhost:Reflector/cluster:X</span></div><div><font class="Apple-style-span" face="Courier"><br></font></div></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">A useful application is a 'confirm' service: when an object is updated in S3 in a region, is the updated version confirmed to be available from clusters in other regions? There is variable lag in consistency in S3 which can cause problems, yet we want to publish a URL to the object as soon as possible.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">To confirm availability, the service publishes a 'poll' message to the global exchange. Agents are listening in an 'admin' vhost in each cluster for this poll and attempt (repetitively if need be) to access the S3 object and check its version. They report back and a consolidated report is returned to the requesting app as JSON.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">Here's a sample printed version – the object was published </span><span class="Apple-style-span" style="font-family: Courier; ">by me using my mac in NYC </span><span style="font-family: Courier; ">to S3 </span><span class="Apple-style-span" style="font-family: Courier; ">in us-east-1 </span><span class="Apple-style-span" style="font-family: Courier; ">via the 'confirm' service running in a vhost in my us-east-1 cluster:</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">Respond</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">Success</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">Tries</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">Elapsed ms</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">Finish/Start</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">Overall</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">1385</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:34.488206Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.103081Z</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">Put to s3</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">227</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.330069Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.103081Z</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">Poll/Response</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">1158</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:34.488206Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.330069Z</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">•us-east-1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">122</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.460626Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.338701Z</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">•ap-northeast-1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">957</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:34.399986Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.442609Z</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">•sa-east-1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">736</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:34.200525Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.464345Z</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">•eu-west-1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">463</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.861053Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.398861Z</span></div><div><span style="font-family: Courier; "><br></span></div><div><span style="font-family: Courier; ">•us-west-1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">true</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">1</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">404</span><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.788365Z</span></div><div><span class="Apple-tab-span" style="white-space: pre; font-family: Courier; "> </span><span style="font-family: Courier; ">2012-07-12T19:29:33.384557Z</span></div></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">An advantage of this approach is that I can run the 'confirm' service in any vhost in any cluster and not have to worry about explicitly federating it across my clusters – I can use this general purpose capability.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">We have many similar low-bandwidth general services in mind, e.g. for monitoring. We'll probably upgrade our hosts to Precision Time Service (PTP) to get more accurate timings.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">Opportunities for improvement:</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">. Static configuration is a pain – my understanding is that this is being remedied. This is most important.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">. Lots of 'extra' messages are sent in my scenario – perhaps this could be optimized. Not too important.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">. Federation within a cluster could perhaps be optimized – actually it is not even documented as a capability. Not too important.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">We have some high volume uses as well for which we will use additional, more tailored federated links. However we are finding this general purpose, low volume capability quite useful both in limited practice and as an architectural concept supporting new services.</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; "><br></span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">Michael Laing</span></div><div style="font-family: Calibri, sans-serif; "><span style="font-family: Courier; ">NYTimes Media Group</span></div></body></html>