[rabbitmq-discuss] Blocked/blocking connections & Memory fluctuations

[shridharan muthu]

Thank you Simon. I have a few questions (inline with the explanation below).


On Thu, Dec 19, 2013 at 7:50 AM, Simon MacMullen <simon at rabbitmq.com> wrote:

> On 18/12/13 17:11, shridharan muthu wrote:
>
>>
>>
>>
>>
>> * Hi there,     I am noticing couple of issues in our rabbit cluster (of
>> 2 nodes). *
>> *  1. I notice a lot of blocked and blocking connections in the
>> management*
>>
>>
>>
>>
>>
>>
>> *     admin UI but "rabbitmqctl list_connections" is not showing any
>> blocked or blocking connections. These connections are living over a
>> day and since we use php in our application layer, none of these
>> connections should not live more than 5 mins. Whats strange is that
>> this symptom is noticed in only one of the nodes of the cluster. *
>>
>
> * Which version are you running? This sounds like a bug that was fixed in
> 3.0.3 where connections and channels might not be cleaned up from the
> management database when their node went down.*
>

       You are spot on Simon. We are using 3.0.2 version in our cluster. In
this case, restarting node will not help I guess. Can I delete these
blocked/blocking connections from Management UI?

>
> *  2. Memory doubles once in a few seconds on same node of the cluster.*
>>
>>
>>
>>
>> *     May be GC is kicking in? Not sure whether it could relate to the
>>   blocked/blocking connections. Since this node didn't hit memory
>> watermark level, logs doesn't have any warning/errors. *
>>
>
> * That sounds like it might be GC, but it's hard to tell.*
>
>
     Could it be cause of the blocked connections? I will check the memory
usage after cleaning up the connections.


>
>>
>>
>>
>>
>>
>> *      On a related note, I have a question about redistributing queues
>> after maintanence. For a cluster with 2 nodes (node 1 & node 2) with
>> mirroring, lets say out of 12 queues, 6 lives in node 1 (as master) and
>> other 6 lives in node 2.  For maintenance reasons, if I take node 2 down
>> and put it back in the cluster, all queues will live node 1 at this point.
>> Is there any way to redistribute the queues among these 2 nodes in this
>> case to move 6 queues back to node 2?*
>>
>
> * You could use a higher priority "nodes" policy to move the master to
> node 2, then delete it again (albeit the queue would be unmirrored while
> this was taking place). Oh, but you need to be on at least RabbitMQ 3.1.x
> for that to work, 3.0.x does not let a mirroring policy move the master.*
>

   Ah! But that would leave with tightly coupling queues to nodes.

>
>
> * OTOH if the queues are mirrored to all nodes anyway, why worry? A slave
> takes about as many resources as a master anyway, so all your masters being
> on one node and all the slaves on the other should be no big deal.*


    My understanding is that all operations (publish, consume) on a queue
will be forwarded from slaves to the master (where it will be executed and
broadcasted to all mirrors) and all the messages in the mirror will be used
only when the master goes down. This sounds like slaves are just forwarding
messages all the time (1 hop penalty) and generates more traffic between
slaves & the master.

>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131219/60397ec7/attachment.html>