No subject
Thu Feb 16 03:44:05 GMT 2012
explicitly specified
--e89a8ff253187c01a204b90c8af0
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Hi Jerry,<br><br>My use case is a typical pub/sub. There is a farm of appli=
cations which broadcast their state to eachother periodically. Each app cre=
ates its own queue and binds it to a well-known broadcast (fanout) exchange=
.<br>
=A0 It is typical for pub/sub to use transient queues.<br>Besides pub/sub b=
roadcast, applications use messaging for critical use cases, such as quick =
cache invalidation, that's why HA is configured.<br><br>> Note that =
you can't straightforwardly redeclare a queue that had been on<br>
> a node that's gone down. =A0The cluster's metadata will still =
know about it<br>> and prevent you from redeclaring it.<br><br>The above=
is the only reason I still declare transient queue an HA. I wouldn't l=
ike it when a failed node cause errors in applications.<br>
<br>Unless you mean that the above applies to a persistent queue only? Othe=
rwise I do not see how transient queue can be used in a cluster environment=
. If a node goes down and there is no way to re-declare a queue with the sa=
me name, it very likely breaks the application tier.<br>
<br>To summarize, <br>=A0 critical use cases -> HA cluster (AKA active/a=
ctive)<br>=A0 pub/sub use case -> transient queues<br>=A0 don't want=
failed node to cause lost queue -> forced to declare transient HA queue=
?<br>
<br>Thanks,<br>Vadim.<br><br><div class=3D"gmail_quote">On Wed, Feb 15, 201=
2 at 2:09 PM, Jerry Kuch <span dir=3D"ltr"><<a href=3D"mailto:jerryk at vmw=
are.com">jerryk at vmware.com</a>></span> wrote:<br><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex">
Hi, Vadim.<br>
<br>
I apologize if I'm misunderstanding you... =A0I'm not entirely sure=
why you'd<br>
want a *transient* queue to be HA. =A0Unless you've developed a mistake=
n<br>
understanding that reading from a queue when you're connected to a part=
icular<br>
cluster node, N, requires a mirror of that queue on the node N: =A0it, in f=
act<br>
does not, and Rabbit will get messages from the queue internally and delive=
r<br>
them to your consumer on whichever node it's connected to.<br>
<br>
So if I'm reading you correctly and your scenario is that you want to d=
eclare<br>
transient queues and access them from any cluster node, then you don't =
have<br>
any extra work to do. =A0Just declare that transient queue, period, and acc=
ess<br>
it freely. =A0You don't have to worry about where it is, and you'd =
use HA only<br>
if you wanted the queue to remain available if the node it lived on went do=
wn.<br>
<br>
Take a look at: =A0<a href=3D"http://www.rabbitmq.com/clustering.html" targ=
et=3D"_blank">http://www.rabbitmq.com/clustering.html</a><br>
<br>
It summarizes what data lives where in a cluster. =A0In particular, that<br=
>
"All data/state required for the operation of a RabbitMQ broker is rep=
licated<br>
across all nodes, for reliability and scaling, with full ACID properties.<b=
r>
An exception to this are message queues, which by default reside on the nod=
e<br>
that created them, though *they are visible and reachable from all nodes*.&=
quot;<br>
<br>
Make sense?<br>
<br>
Best regards,<br>
Jerry<br>
<br>
<br>
<br>
~~~~]<br>
:<br>
<div class=3D"im HOEnZb"><br>
----- Original Message -----<br>
From: "Vadim Chekan" <<a href=3D"mailto:kot.begemot at gmail.com"=
>kot.begemot at gmail.com</a>><br>
</div><div class=3D"HOEnZb"><div class=3D"h5">To: "Jerry Kuch" &l=
t;<a href=3D"mailto:jerryk at vmware.com">jerryk at vmware.com</a>><br>
Cc: <a href=3D"mailto:rabbitmq-discuss at lists.rabbitmq.com">rabbitmq-discuss=
@lists.rabbitmq.com</a><br>
Sent: Wednesday, February 15, 2012 12:11:45 PM<br>
Subject: Re: [rabbitmq-discuss] Active/Active: shutdown of one service brin=
gs down the cluster<br>
<br>
Hi Jerry,<br>
<br>
So is there a better way to declare transient queues then declaring them as=
HA?<br>
I can see only alternative by adding a random string to queue name. And whi=
ch way is preferred?<br>
<br>
Vadim.<br>
<br>
<br>
On Tue, Feb 14, 2012 at 12:45 PM, Jerry Kuch < <a href=3D"mailto:jerryk@=
vmware.com">jerryk at vmware.com</a> > wrote:<br>
<br>
<br>
Hi, Vadim:<br>
<br>
A client doesn't need to be connected directly to the node on which<br>
a queue and its attendant Erlang process reside. If your load balancer<br>
sends you to any live node in the cluster you can consume from the queue<br=
>
of your choice, as long as it's still alive.<br>
<br>
Note that you can't straightforwardly redeclare a queue that had been o=
n<br>
a node that's gone down. The cluster's metadata will still know abo=
ut it<br>
and prevent you from redeclaring it. This is intentional, to avoid the<br>
confusion that would result if you succeeded at the redeclare, a new<br>
queue of the same name and properties came into existence on another node,<=
br>
and then the original, downed node came back up in the cluster...<br>
<br>
Best regards,<br>
Jerry<br>
<br>
<br>
<br>
----- Original Message -----<br>
From: "Vadim Chekan" < <a href=3D"mailto:kot.begemot at gmail.com=
">kot.begemot at gmail.com</a> ><br>
To: "Simon MacMullen" < <a href=3D"mailto:simon at rabbitmq.com">=
simon at rabbitmq.com</a> >, <a href=3D"mailto:ghanna at verticalsearchworks.c=
om">ghanna at verticalsearchworks.com</a><br>
Cc: <a href=3D"mailto:rabbitmq-discuss at lists.rabbitmq.com">rabbitmq-discuss=
@lists.rabbitmq.com</a><br>
Sent: Tuesday, February 14, 2012 12:40:05 PM<br>
Subject: Re: [rabbitmq-discuss] Active/Active: shutdown of one service brin=
gs down the cluster<br>
<br>
<br>
Hi Simon,<br>
<br>
Thanks for looking into the logs. Since we fixed channel leak in our applic=
ation we do not experience any problems anymore.<br>
<br>
Regarding transient queues in HA. I am just not sure how system would behav=
e when non-HA queue is declared in a cluster environment. Documentation des=
cribes in great details what happen to mirrored queues but I can't find=
anything about non-ha queue in HA cluster. Queue will be created on a sing=
le server, and application should be ready to re-declare queue in case of f=
ailover. So far so good. But how does it work with load balancer? When requ=
est is made against a server which does not have a given queue, will the cl=
uster "know" where the queue is and proxy the request to the prop=
er server?<br>
<br>
Thanks,<br>
Vadim.<br>
<br>
<br>
On Mon, Feb 13, 2012 at 10:09 AM, Simon MacMullen < <a href=3D"mailto:si=
mon at rabbitmq.com">simon at rabbitmq.com</a> > wrote:<br>
<br>
<br>
<br>
On 10/02/12 00:20, Vadim Chekan wrote:<br>
<br>
<br>
I think we nailed down a problem. We had a channel leak in our<br>
application. With ~50 connections we had >90 channels per connection and=
<br>
growing. This definitely correlates to high CPU usage.<br>
<br>
What I still do not understand either it triggered rabbit into unstable<br>
state or it was something else. Maybe increasing latencies in message<br>
handling triggered cluster members into flipping neighbor aliveness<br>
status back and force? Just speculating here: could timeouts because of<br>
high load cause network fragmentation, when every node temporally does<br>
not see neighbors, becomes a master, than see a neighbor, freak out, etc?<b=
r>
<br>
That's plausible, but I don't think that's what's happening=
(there's nothing about network partitioning in the logs).<br>
<br>
<br>
<br>
<br>
I've attached logs from all 3 cluster members. They are polluted with<b=
r>
load balancer "ping".<br>
<br>
Thanks. I've had a poke at this but nothing is leaping out at me yet. I=
'll keep at it though.<br>
<br>
One thing that's a bit odd: you seem to be creating HA / transient / au=
todelete / exclusive queues. So although they're "HA", they w=
ill vanish if any of the following happens:<br>
<br>
* The entire cluster goes down (transient) or<br>
* All consumers for a queue cancel (autodelete) or<br>
* The connection that created them closes (exclusive)<br>
<br>
Is this intentional? It seems like an odd use of HA.<br>
<br>
Cheers, Simon<br>
<br>
<br>
<br>
--<br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
<br>
<br>
<br>
--<br>
More information about the rabbitmq-discuss
mailing list