[rabbitmq-discuss] Blocked/blocking connections & Memory fluctuations
shridharan muthu
shridharan.m at gmail.com
Mon Dec 23 16:44:05 GMT 2013
Hello Simon,
I ended up restarting the node. Here is what I saw from the logs
during management restart.
*rabbit_mochiweb_registry crash:*
*=ERROR REPORT==== 19-Dec-2013::11:50:28 ===*
*** Generic server rabbit_mochiweb_registry terminating*
*** Last message in was {add,rabbit_mgmt_redirect,*
* [{port,55672},{ignore_in_use,true}],*
* #Fun<rabbit_mochiweb.1.105980535>,*
* #Fun<rabbit_mochiweb.0.34137573>,*
* {[],"Redirect to port 15672"}}*
*** When Server state == undefined*
*** Reason for termination ==*
*** {{case_clause,*
*
{error,{no_record_for_listener,[{port,55672},{ignore_in_use,true}]}}},*
* [{rabbit_mochiweb_registry,handle_call,3},*
* {gen_server,handle_msg,5},*
* {proc_lib,init_p_do_apply,3}]}*
*=INFO REPORT==== 19-Dec-2013::11:50:28 ===*
* application: rabbitmq_management*
* exited: {bad_return,*
* {{rabbit_mgmt_app,start,[normal,[]]},*
* {'EXIT',*
* {{{case_clause,*
* {error,*
* {no_record_for_listener,*
* [{port,55672},{ignore_in_use,true}]}}},*
* [{rabbit_mochiweb_registry,handle_call,3},*
* {gen_server,handle_msg,5},*
* {proc_lib,init_p_do_apply,3}]},*
* {gen_server,call,*
* [rabbit_mochiweb_registry,*
* {add,rabbit_mgmt_redirect,*
* [{port,55672},{ignore_in_use,true}],*
* #Fun<rabbit_mochiweb.1.105980535>,*
* #Fun<rabbit_mochiweb.0.34137573>,*
* {[],"Redirect to port 15672"}},*
* infinity]}}}}}*
* type: temporary*
On Fri, Dec 20, 2013 at 4:22 AM, Simon MacMullen <simon at rabbitmq.com> wrote:
> Hmm. I cannot figure out how that could come to pass - it looks like there
> is a broken listener already there.
>
> You might be able to fix this by invoking
>
> $ rabbitmqctl -n rabbit eval 'rabbit_web_dispatch_sup:stop_
> listener([{port,55672},{ignore_in_use,true}]).'
>
> but if that doesn't work I think you'd have to restart the node I'm afraid.
>
> Either way, I'd be very interested to see the logs from around when you
> tried to restart the management application.
>
> Cheers, Simon
>
>
> On 19/12/13 19:38, shridharan muthu wrote:
>
>> Hello Simon,
>>
>> I tried restarting rabbitmq management and I got the following
>> error. I am not able to use port 15672 at all.
>>
>> /sudo rabbitmqctl -n rabbit eval
>> 'application:stop(rabbitmq_management),application:start(
>> rabbitmq_management).'/
>> /{error,/
>> / {bad_return,/
>> / {{rabbit_mgmt_app,start,[normal,[]]},/
>> / {'EXIT',/
>> / {{{case_clause,/
>> / {error,/
>> / {no_record_for_listener,/
>> / [{port,55672},{ignore_in_use,true}]}}},/
>> / [{rabbit_mochiweb_registry,handle_call,3},/
>> / {gen_server,handle_msg,5},/
>> / {proc_lib,init_p_do_apply,3}]},/
>> / {gen_server,call,/
>> / [rabbit_mochiweb_registry,/
>> / {add,rabbit_mgmt_redirect,/
>> / [{port,55672},{ignore_in_use,true}],/
>> / #Fun<rabbit_mochiweb.1.105980535>,/
>> / #Fun<rabbit_mochiweb.0.34137573>,/
>> / {[],"Redirect to port 15672"}},/
>> / infinity]}}}}}}/
>> /...done./
>> /
>>
>> /
>> So, Is there anyway bring it back up without restarting the app?
>>
>> Shri
>>
>>
>> On Thu, Dec 19, 2013 at 9:27 AM, Simon MacMullen <simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>> wrote:
>>
>> On 19/12/13 16:35, shridharan muthu wrote:
>>
>> You are spot on Simon. We are using 3.0.2 version in our
>> cluster. In this case, restarting node will not help I guess. Can
>> I
>> delete these blocked/blocking connections from Management UI?
>>
>>
>> The only way to do that is to restart the management stats database:
>>
>> # rabbitmqctl -n <node with the stats db> eval
>> 'application:stop(rabbitmq___management),application:start(
>> __rabbitmq_management).'
>>
>>
>>
>> Could it be cause of the blocked connections? I will
>> check the
>> memory usage after cleaning up the connections.
>>
>>
>> I doubt it, those connections don't really exist except as rows in
>> the management stats db. But note that we have fixed a few memory
>> leaks since 3.0.2.
>>
>> /You could use a higher priority "nodes" policy to move the
>> master
>>
>> to node 2, then delete it again (albeit the queue would be
>> unmirrored while this was taking place). Oh, but you need
>> to be on
>> at least RabbitMQ 3.1.x for that to work, 3.0.x does not let
>> a
>> mirroring policy move the master./
>>
>>
>> Ah! But that would leave with tightly coupling queues to
>> nodes.
>>
>>
>> Only until you delete the "nodes" policy - you can use that to move
>> the master, but then revert to an "all" policy; at which point the
>> master will stay where it is.
>>
>> /OTOH if the queues are mirrored to all nodes anyway, why
>> worry? A
>>
>> slave takes about as many resources as a master anyway, so
>> all your
>> masters being on one node and all the slaves on the other
>> should be
>> no big deal.
>> /
>>
>> //
>>
>> My understanding is that all operations (publish, consume)
>> on a
>> queue will be forwarded from slaves to the master (where it will
>> be
>> executed and broadcasted to all mirrors) and all the messages in
>> the
>> mirror will be used only when the master goes down.
>>
>>
>> Yes.
>>
>>
>> This sounds like
>> slaves are just forwarding messages all the time (1 hop penalty)
>> and
>> generates more traffic between slaves & the master.
>>
>>
>> Sure, but how will distributing the masters around the cluster help
>> that? Or are you saying your clients know that some queues are on
>> node 2, and expect to connect to that node to use that queue? If so
>> that makes sense...
>>
>>
>> Cheers, Simon
>>
>> --
>> Simon MacMullen
>> RabbitMQ, Pivotal
>>
>>
>>
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131223/70da9667/attachment.html>
More information about the rabbitmq-discuss
mailing list