[rabbitmq-discuss] mirrored queue cluster

[Laing, Michael]

We run mirrored queues within multiple parallel clusters in the 'backend'.

Internal apps are load-balanced into these 'pipelines'.

'Frontend' rabbits consume from each pipeline, and serve client apps via
load balancers, and our own gateway.

The 'frontends' autoscale out, depending upon actual load and predicted
load.

The 'backends' are first scaled up (larger boxes) and then more pipelines
are added (scale out).

One piece of advice for performance: do not use persistence. We use
Cassandra for that purpose: replica resolution, journaling, etc., and can
continuously run analysis a couple minutes after the fact that shows
latencies, outages, and so forth across the entire fabric from an
application point of view.

In fact our project is called 'nyt⨍aбrik'.

Our deployment spans multiple AWS regions, multiple zones per region, and
is headless.

We're currently transitioning into production so we don't have much time to
discuss, but we will do comprehensive blog posts, github projects etc about
the architecture including detailed configurations, frameworks, and so
forth later this year/early next. Almost all components will be
open-sourced.

Michael




On Fri, Oct 4, 2013 at 10:48 AM, Michael Klishin <michael at rabbitmq.com>wrote:

>
> On oct 4, 2013, at 6:40 p.m., "PATAR, SAGAR" <sp345s at att.com> wrote:
>
> > Below when u say queues ..do you mean Exchanges ??
>
> When I say "queues" I mean "queues".
>
> > Do you suggest to have separate queues per consuming system (dynamically
> by the consumer ??)
>
> I'm not sure what "consuming system" means but yes, if you can use
> multiple queues in
> some cases, do that. E.g. if you are streaming log events and have 5
> severities,
> you can use a Single Giant Queue™ (SGQ) or 5 smaller queues, one per
> severity. 5 smaller
> queues will both utilize your hardware resources better and will let you
> distributed
> the load across nodes.
>
> SGQ is not a great design decision in general, but especially so for
> loaded systems.
>
> Note that it's perfectly fine and not uncommon for a single application or
> instance
> to consume from multiple queues.
>
> MK
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/e5392c2a/attachment.htm>