<div dir="ltr">Thanks for the detail! Very interesting.<div><br></div><div>I'm even more curious about the client connection via the DNS name. Isn't that essentially the same as exposing a virtual IP to the client and load balancing across the nodes in the cluster? Or did I misunderstand and the client is routed to different clusters...</div>
<div><br></div><div>Cheers,</div><div><br></div><div>-ronc</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 12, 2014 at 11:12 AM, Laing, Michael <span dir="ltr"><<a href="mailto:michael.laing@nytimes.com" target="_blank">michael.laing@nytimes.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">All of our inter-cluster connections use shovels, both within and between regions.<div><br></div><div>A cluster picks one of its nodes to run the shovel on. That node takes the configured list of nodes in a remote cluster and picks one to connect to. When local or remote nodes go down, things adjust. Mostly we see this during rolling restarts. We have found it very rugged in production.</div>
<div><br></div><div>External clients connect via a DNS name which will round-robin to one of the cluster nodes. We use Route 53 health checks to insure nodes are in service.</div><div><br></div><div>Our external clients use PHP, Java, node.js, and whatever else to connect - possibly some of them are using clients smart enough to fail over by themselves... so we also expose the DNS name of each node in the cluster as an option.</div>
<div><br></div><div>Best,</div><div><br></div><div>Michael</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 12, 2014 at 1:20 PM, Ron Cordell <span dir="ltr"><<a href="mailto:ron.cordell@gmail.com" target="_blank">ron.cordell@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Michael,<div><br></div><div>Thanks for the response - that's very interesting. We were quite interested in your setup when you posted to the rabbit list about the setup for the NYT :)</div>
<div><br></div>
<div>How exactly do you distribute the connections? Does the rabbit driver do that for you by choosing from a list, or do you use some other method?</div><div><br></div><div>Cheers,</div><div><br></div><div>Ron</div></div>
<div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Feb 11, 2014 at 4:05 PM, Laing, Michael <span dir="ltr"><<a href="mailto:michael.laing@nytimes.com" target="_blank">michael.laing@nytimes.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That's interesting!<div><br></div><div>We have removed all the load balancers from our core configurations in Amazon EC2 because we found they added no value, and, in fact provided troublesome additional points of failure. (We do use ELBs to find websocket endpoints in the client-facing retail layer)</div>
<div><br></div><div>Our core clusters in Oregon and Dublin each have 50 - 100 non-local connections, randomly distributed, and are very stable.</div><div><br></div><div>We use DNS with health checks for internal client connections in lieu of load balancers. Simple and rugged.</div>
<span><font color="#888888">
<div><br></div><div>Michael Laing</div><div>NYTimes</div></font></span></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Tue, Feb 11, 2014 at 6:42 PM, Ron Cordell <span dir="ltr"><<a href="mailto:ron.cordell@gmail.com" target="_blank">ron.cordell@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi all --<div><br></div><div>We've been performance testing RabbitMQ on Linux as we're about to move our RabbitMQ infrastructure from Windows to Linux (as well as other things). I wanted to share some of what we observed and if people have any feedback. All tests were done using a 3-node cluster where most queues are HA, with an F5 configured to provide a virtual IP to the application. There is a single vHost.</div>
<div><br></div><div>1. On the same hardware the Linux installation easily outperforms the Windows installation. It also uses fewer resources for the same throughput.</div><div><br></div><div>2. The Windows cluster becomes unstable and nodes start dropping out/partitioning at around 1/3 max tested volume. The Linux cluster showed no instability whatsoever up to maximum throughput.</div>
<div><br></div><div>3. Creating a cluster with 2 RAM nodes and 1 Disc node has the same disk I/O requirements as 3 disc nodes. (This makes sense because as I believe the RAM nodes will persist to disk for HA queues).</div>
<div><br></div><div>4. (here is the interesting one) When the F5 is configured to load balance across the 3 nodes as a round-robin load balancer, maximum throughput is significantly less than if the F5 sends all traffic to a single node.</div>
<div><br></div><div>I'd love any feedback, especially on #4.</div><div><br></div><div>Cheers!</div><div><br></div><div>-ronc</div></div>
<br></div></div><div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></div></blockquote></div><br></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>