<div dir="ltr">We run mirrored queues within multiple parallel clusters in the 'backend'.<div><br></div><div>Internal apps are load-balanced into these 'pipelines'.</div><div><br></div><div>'Frontend' rabbits consume from each pipeline, and serve client apps via load balancers, and our own gateway.</div>
<div><br></div><div>The 'frontends' autoscale out, depending upon actual load and predicted load.</div><div><br></div><div>The 'backends' are first scaled up (larger boxes) and then more pipelines are added (scale out).</div>
<div><br></div><div>One piece of advice for performance: do not use persistence. We use Cassandra for that purpose: replica resolution, journaling, etc., and can continuously run analysis a couple minutes after the fact that shows latencies, outages, and so forth across the entire fabric from an application point of view.</div>
<div><br></div><div>In fact our project is called '<span class="">nyt</span>⨍aбrik'.</div>
<div><br></div><div>Our deployment spans multiple AWS regions, multiple zones per region, and is headless.</div><div><br></div><div>We're currently transitioning into production so we don't have much time to discuss, but we will do comprehensive blog posts, github projects etc about the architecture including detailed configurations, frameworks, and so forth later this year/early next. Almost all components will be open-sourced.</div>
<div><br></div><div>Michael<br><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Oct 4, 2013 at 10:48 AM, Michael Klishin <span dir="ltr"><<a href="mailto:michael@rabbitmq.com" target="_blank">michael@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
On oct 4, 2013, at 6:40 p.m., "PATAR, SAGAR" <<a href="mailto:sp345s@att.com">sp345s@att.com</a>> wrote:<br>
<br>
> Below when u say queues ..do you mean Exchanges ??<br>
<br>
</div>When I say "queues" I mean "queues".<br>
<div class="im"><br>
> Do you suggest to have separate queues per consuming system (dynamically by the consumer ??)<br>
<br>
</div>I'm not sure what "consuming system" means but yes, if you can use multiple queues in<br>
some cases, do that. E.g. if you are streaming log events and have 5 severities,<br>
you can use a Single Giant Queue™ (SGQ) or 5 smaller queues, one per severity. 5 smaller<br>
queues will both utilize your hardware resources better and will let you distributed<br>
the load across nodes.<br>
<br>
SGQ is not a great design decision in general, but especially so for loaded systems.<br>
<br>
Note that it's perfectly fine and not uncommon for a single application or instance<br>
to consume from multiple queues.<br>
<span class="HOEnZb"><font color="#888888"><br>
MK<br>
<br>
<br>
<br>
</font></span><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>