<div dir="ltr">Thank you Simon. I have a few questions (inline with the explanation below).<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 19, 2013 at 7:50 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">On 18/12/13 17:11, shridharan muthu wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im"><i>
Hi there,<br>
<br>
I am noticing couple of issues in our rabbit cluster (of 2 nodes).<br>
<br></i></div><i>
1. I notice a lot of blocked and blocking connections in the management</i><div class="im"><i><br>
admin UI but "rabbitmqctl list_connections" is not showing any<br>
blocked or blocking connections. These connections are living over a<br>
day and since we use php in our application layer, none of these<br>
connections should not live more than 5 mins. Whats strange is that<br>
this symptom is noticed in only one of the nodes of the cluster.<br>
</i></div></blockquote>
<i><br>
Which version are you running? This sounds like a bug that was fixed in 3.0.3 where connections and channels might not be cleaned up from the management database when their node went down.</i><br></blockquote><div> </div>
<div> You are spot on Simon. We are using 3.0.2 version in our cluster. In this case, restarting node will not help I guess. Can I delete these blocked/blocking connections from Management UI? </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<i><br>
</i><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><i>
2. Memory doubles once in a few seconds on same node of the cluster.</i><div class="im"><i><br>
May be GC is kicking in? Not sure whether it could relate to the<br>
blocked/blocking connections. Since this node didn't hit memory<br>
watermark level, logs doesn't have any warning/errors.<br>
</i></div></blockquote>
<i><br>
That sounds like it might be GC, but it's hard to tell.</i><div class="im"> <br></div></blockquote><div> Could it be cause of the blocked connections? I will check the memory usage after cleaning up the connections. </div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><i>
On a related note, I have a question about redistributing queues<br>
after maintanence. For a cluster with 2 nodes (node 1 & node 2) with<br>
mirroring, lets say out of 12 queues, 6 lives in node 1 (as master) and<br>
other 6 lives in node 2. For maintenance reasons, if I take node 2 down<br>
and put it back in the cluster, all queues will live node 1 at this<br>
point. Is there any way to redistribute the queues among these 2 nodes<br>
in this case to move 6 queues back to node 2?</i><br>
</blockquote>
<i><br></i></div><i>
You could use a higher priority "nodes" policy to move the master to node 2, then delete it again (albeit the queue would be unmirrored while this was taking place). Oh, but you need to be on at least RabbitMQ 3.1.x for that to work, 3.0.x does not let a mirroring policy move the master.</i><br>
</blockquote><div> </div><div> Ah! But that would leave with tightly coupling queues to nodes.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br><i>
OTOH if the queues are mirrored to all nodes anyway, why worry? A slave takes about as many resources as a master anyway, so all your masters being on one node and all the slaves on the other should be no big deal.<br></i></blockquote>
<div><i> </i></div><div> My understanding is that all operations (publish, consume) on a queue will be forwarded from slaves to the master (where it will be executed and broadcasted to all mirrors) and all the messages in the mirror will be used only when the master goes down. This sounds like slaves are just forwarding messages all the time (1 hop penalty) and generates more traffic between slaves & the master. </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Cheers, Simon<span class=""><font color="#888888"><br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</font></span></blockquote></div><br></div></div>