[rabbitmq-discuss] RabbitMQ HA with Redis

Thu Apr 15 15:38:11 BST 2010

Alexis,

On Thu, Apr 15, 2010 at 7:51 AM, Alexis Richardson <
alexis.richardson at gmail.com> wrote:

> Gustavo,
>
> On Wed, Apr 14, 2010 at 9:35 PM, Gustavo Aquino
> <aquino.gustavo at gmail.com> wrote:
> > Alexis,
> > Why not one HA OOTB can be provided ?
>
> As I mentioned, we are about to release an HA Guide which provides our
> recommendation for HA today, that we can stand behind.  In other words
> - you can trust us to commercially support that solution and provide
> SLAs.
>
>
I trust you, you know, but we need to know what will be the road map for it.

> You are correct that we *could* provide a single configuration OOTB
> for HA.  The question is: which one.
>
> Bear in mind that HA behaviour combines:
>
> * Immediate acceptance
> * Immediate delivery
> * Eventual acceptance
> * Eventual delivery
>
> Note that:
>
> (a) these requirements are in tension with one another; so it is
> expensive to provide a system that is able to provide all of them at
> once so that, somehow, the system works out what each message needs
> and reacts accordingly based on need.
>
> (b) more importantly, the requirements are based on a notion of TIME.
> This is problematic because time based guarantees are
> non-deterministic.  For some systems an unavailability window of 1ms
> is fine because the client can wait.  For others it is too big.  Many
> things can affect this.  For example you could be running a messaging
> system on a real time Linux kernel, but still have variance in your
> failover window due to how your network and disk hardware is behaving
> based on (eg) load and contention.
>
> This is all ok, but we need to make our assumptions *explicit* for any
> one HA OOTB solution.
>
>
>
You are right. But some situations are the same of ppl, I think that a good
solution is We have this ways to do HA OOTB it's part of product, but if you
not attend you we can optimize it soing something like X,Y or
Z approaches... (your commercial support).

>
> > do you have one HP OOTB solution why
> > not HA ? May be one HA OOTB is not the best solution, and may be not
> attend
> > 100% of users but may be 70% and it's sounds good for me not for you ?.
>
> Yes that's fine Gustavo, but as soon as you make a promise about one
> configuration being "HA" then people immediately expect that you can
> do things that break the laws of physics.

> > We know about one AMQP product that have a HA OOTB solution, and we are
> > trying this "HA OOTB" and this product will be used in CME. But can't do
> the
> > same with RabbitMQ.
>
> Why don't you describe the main HA parameters that you need, on this
> list, and we can talk about them.
>
>
Sure, here is.

We have two scenarios for HA:
-  Cluster
- "Federation"

Cluster:

A* Cluster* will be a group of brokers that act as a single broker. Changes
on any broker are replicated to all other brokers in the same Messaging
Cluster, so if one broker fails, its clients can fail-over to another broker
without loss of state. The brokers in a Messaging Cluster may run on the
same host or on different hosts.
Clusters need to provide a reliable multicast protocol, tools, and
infrastructure for implementing replicated services.
Clusters need to uses an Active/Active model. In this model, all possible
brokers are active at all times. Producers and consumers can be connected to
any broker in the cluster. Additionally, any broker can be killed and
restarted, and the cluster will retain its operational state to provide
scalability and enhanced load-balancing.

"Federation":

Federation will be used to provide geographical distribution of brokers. A
number of individual brokers, or clusters of brokers, can be federated
together. This will allows client machines to see and interact with the
federation as though it were a single broker. Federation will be used where
client machines remain on a local network, even though their messages have
to be routed out.

Federation will be used for connecting disparate locations across a wide
area network. Full connectivity across an enterprise can be achieved while
keeping local message traffic isolated to a single location. Departmental
brokers can be specified with individual policies that control
inter-departmental message traffic flow.
Client produces data that must be delivered reliably but connectivity can
not be guaranteed. In this case, a co-resident broker provides queueing and
durability that is not available in the client on its own.

RabbitMQ provide a part of "Federation" but the problem is node dow all
others don't see more the queue and it broke all the process, and RabbitMQ
"route" don't guarantee delivery if one node is down out of local network,
today if one node is down all process fail.

> Also: Do you want to see a copy of the HA Guide?
>

I would like so much to get a copy of this guide.

> > I understand you, and you know my case, we are working in
> > microseconds/nanoseconds, may be this HA don't attend our requirements,
> but
> > it is one solutions and until now we don't have a good solution for HA
> using
> > Rabbit, about HP we don't need nothing to talk about RabbitMQ is very
> good
> > Broker routing 2.000.000 messages per second just inboud. But one good HA
> > solution is getting my sleep.
>
> I'm sorry to hear that we are keeping you awake at night.
>

No worry it's not your fault, it's part of IT skill :)

>
>
> > Do you know what we need,we talked about that is just one HA solution
> from
> > RabbitMQ with a very low latency, SLA and guarantee that no messages will
> be
> > loosed. O.S cluster or putting HA logic in our application is not a good
> > solution in our case.
>
> Actually, doing dedup on the consumer could be a very good solution
> based on what you have told me in the past.  But, let's keep
> discussing it.
>
> alexis
>

Great. let's discussing about.

Regards.

Gustavo

>
>
>
> > Regards.
> >
> >
> >
> > On Wed, Apr 14, 2010 at 4:56 PM, Alexis Richardson
> > <alexis.richardson at gmail.com> wrote:
> >>
> >> Gustavo
> >>
> >> We are about to release an HA Guide including recommendations.  It's
> >> just being reviewed.
> >>
> >> It is not possible to provide "HA OOTB" because each system is
> >> slightly different.  If someone tells you that they can support HA
> >> OOTB, they are misleading you.  What is possible, and desirable, is a
> >> much simpler and more "OOTB like" experience for the main four or five
> >> HA cases.  For example, the Beetle project aims to solve for cases
> >> where idempotency is desired.  Your case might require a different
> >> optimisation.  E.g. sometimes latency needs to be below X
> >> microseconds, and sometimes it does not.
> >>
> >> We are in the business of providing long term commercial support for
> >> RabbitMQ based solutions, including where an SLA is in use.  Just let
> >> us know what solution or solutions you need.
> >>
> >> alexis
> >>
> >>
> >>
> >> On Wed, Apr 14, 2010 at 8:43 PM, Gustavo Aquino
> >> <aquino.gustavo at gmail.com> wrote:
> >> > Alexis,
> >> > I know that it's not the unique, you know how much I'm looking for a
> >> > good HA
> >> > solutions using RabbitMQ. The problem is RabbitMQ doesn't have a
> default
> >> > solution out of the box for HA, and do you know companies need it,
> need
> >> > a
> >> > commitment from product owner about HA. Do my own solution means,
> >> > the responsible of maintenance and evolution of this solutions is my,
> >> > and it
> >> > is not my business it will be yours. Understand why I do this question
> >> > about
> >> > support ?
> >> > I'm a community enthusiast, and work hard of lot of open projects and
> is
> >> > a
> >> > pleasure if I can help to Rabbit HA, the point is today all products
> >> > inclusive cloud products need to have one HA solutions OTB, It's is
> the
> >> > best
> >> > ? may be not but need to have it, and today we have a problem of using
> >> > RabbitMQ because it don't have it. So how can I put Rabbit on my core
> if
> >> > cannot guarantee a single SLA, or without putting responsible in side
> >> > like
> >> > O.S cluster solution, OpenaIS, LVS shared storage or whatever.
> >> > Do you know any about the overhead in throughput using this Xing's
> >> > Solution
> >> > ?
> >> >
> >> > Best Regards.
> >> >
> >> > On Wed, Apr 14, 2010 at 4:08 PM, Alexis Richardson
> >> > <alexis.richardson at gmail.com> wrote:
> >> >>
> >> >> Thanks Salvatore.
> >> >>
> >> >> Gustavo,
> >> >>
> >> >> We and the community will support good solutions and 'things that
> >> >> work'.  I like the approach, but as Salvatore suggested - it is not
> >> >> unique.
> >> >>
> >> >> alexis
> >> >>
> >> >>
> >> >>
> >> >> On Wed, Apr 14, 2010 at 7:37 PM, Salvatore Sanfilippo
> >> >> <antirez at gmail.com>
> >> >> wrote:
> >> >> > On Wed, Apr 14, 2010 at 8:34 PM, Gustavo Aquino
> >> >> > <aquino.gustavo at gmail.com> wrote:
> >> >> >
> >> >> >> So if RabbitMQ is a VMWare product and Redis it too, so do Rabbit
> >> >> >> support
> >> >> >> this HA approach ? :-)
> >> >> >
> >> >> > I think that surely it helps that this feature depends from another
> >> >> > VMWare product ;)
> >> >> > But my guess is that the way to go is to have such a support
> built-in
> >> >> > for RabbitMQ.
> >> >> >
> >> >> > Ciao,
> >> >> > Salvatore
> >> >> >
> >> >> >> Regars.
> >> >> >>
> >> >> >> On Wed, Apr 14, 2010 at 2:09 PM, Jason J. W. Williams
> >> >> >> <jasonjwwilliams at gmail.com> wrote:
> >> >> >>>
> >> >> >>> Been reading the page since y'all started tweeting about it. :)
> >> >> >>> It's
> >> >> >>> hard to tell how it works.
> >> >> >>>
> >> >> >>> -J
> >> >> >>>
> >> >> >>> On Wed, Apr 14, 2010 at 11:02 AM, Alexis Richardson
> >> >> >>> <alexis.richardson at gmail.com> wrote:
> >> >> >>> > Hi everyone,
> >> >> >>> >
> >> >> >>> > It looks like Xing's Beetle project is up on Github:
> >> >> >>> > http://github.com/xing/beetle
> >> >> >>> >
> >> >> >>> > There is a project page here: http://xing.github.com/beetle/
> >> >> >>> >
> >> >> >>> > Please take a look and share any comments you may have here :-)
> >> >> >>> >
> >> >> >>> > alexis
> >> >> >>> >
> >> >> >>> > _______________________________________________
> >> >> >>> > rabbitmq-discuss mailing list
> >> >> >>> > rabbitmq-discuss at lists.rabbitmq.com
> >> >> >>> >
> >> >> >>> >
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >> >> >>> >
> >> >> >>>
> >> >> >>> _______________________________________________
> >> >> >>> rabbitmq-discuss mailing list
> >> >> >>> rabbitmq-discuss at lists.rabbitmq.com
> >> >> >>>
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Salvatore 'antirez' Sanfilippo
> >> >> > http://invece.org
> >> >> >
> >> >> > "Once you have something that grows faster than education grows,
> >> >> > you’re always going to get a pop culture.", Alan Kay
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100415/606ec09a/attachment.htm 