[rabbitmq-discuss] Unexaplainable behaviour with shovel plugin.

[Laing, Michael]

On Wed, Mar 5, 2014 at 1:32 AM, Ben Hood <0x6e6562 at gmail.com> wrote:

> On Sun, Mar 2, 2014 at 3:21 AM, Laing, Michael
> <michael.laing at nytimes.com> wrote:
> > Persistence might increase reliability when you plan to restart nodes and
> > need to regain state. We don't do that.
>
> Are you referring to Rabbit nodes heres? Reading further into your
> description, it looks like the application state is maintained by
> Cassandra replication.
>

Yes. Rabbit nodes.


>
> > We target queue lengths of zero and are close most of the time. Anything
> > else stands out like a black spot on a white sheet.
>
> How do you achieve this steady state? Have you calibrated your app in
> some fashion? Or do you apply flow control?
>

Some of our fabrik apps apply flow control. For example, we apply flow
control to the ingress of 'silver' service messages so that 'gold' service
takes precedence.

Since our services are fully parallelized, if a bottleneck appears, we add
more workers to that stage, distributed across the rabbit cluster. For
example, our 'cache_pull' service has 9 workers in each core pipeline.

I think SEDA is a good inspiration for this kind of design - Cassandra is
organized internally this way - nyt⨍aбrik is SEDA writ large with bigger
chunks. Here's a SEDA link from the originator:
http://matt-welsh.blogspot.com/2010/07/retrospective-on-seda.html


>
> > So we never restart nodes that die. Just sync in new ones. Actually we
> have
> > not yet had any core nodes die in production.
>
> Again - you're talking about Rabbit nodes right?
>

Yup.


>
> > Our instances are ridiculously small and inexpensive to run. We rely on
> this
> > global, headless, mutually supporting rabbit army for our reliability,
> > paired with a small Cassandra horde.
>
> Seems like a simple yet effective approach given the scale you're
> targeting.
>

Actually I think it will work for anyone that wants a global, reliable,
public-facing bi-directional messaging platform that will scale up when it
needs, down when it doesn't, and is presented as a standard service to
internal customers. We're the ham in the sandwich - and much more reliable
than the bread slices.

The back-end is 'pluggable' - couchbase might be an alternate choice. We've
used riak and dynamodb but chose Cassandra because multi-datacenter
facilities are available and free to use plus we have folks who can jump
into any Java issues (not me).


> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140305/0bd855c5/attachment.html>