<div dir="ltr">Hello Simon,<div> </div><div> I ended up restarting the node. Here is what I saw from the logs during management restart.</div><div><div><br></div><div><i>rabbit_mochiweb_registry crash:</i></div><div>
<i>=ERROR REPORT==== 19-Dec-2013::11:50:28 ===</i></div><div><i>** Generic server rabbit_mochiweb_registry terminating</i></div><div><i>** Last message in was {add,rabbit_mgmt_redirect,</i></div><div><i> [{port,55672},{ignore_in_use,true}],</i></div>
<div><i> #Fun<rabbit_mochiweb.1.105980535>,</i></div><div><i> #Fun<rabbit_mochiweb.0.34137573>,</i></div><div><i> {[],"Redirect to port 15672"}}</i></div>
<div><i>** When Server state == undefined</i></div><div><i>** Reason for termination ==</i></div><div><i>** {{case_clause,</i></div><div><i> {error,{no_record_for_listener,[{port,55672},{ignore_in_use,true}]}}},</i></div>
<div><i> [{rabbit_mochiweb_registry,handle_call,3},</i></div><div><i> {gen_server,handle_msg,5},</i></div><div><i> {proc_lib,init_p_do_apply,3}]}</i></div><div><i><br></i></div><div><i>=INFO REPORT==== 19-Dec-2013::11:50:28 ===</i></div>
<div><i> application: rabbitmq_management</i></div><div><i> exited: {bad_return,</i></div><div><i> {{rabbit_mgmt_app,start,[normal,[]]},</i></div><div><i> {'EXIT',</i></div><div>
<i> {{{case_clause,</i></div><div><i> {error,</i></div><div><i> {no_record_for_listener,</i></div><div><i> [{port,55672},{ignore_in_use,true}]}}},</i></div>
<div><i> [{rabbit_mochiweb_registry,handle_call,3},</i></div><div><i> {gen_server,handle_msg,5},</i></div><div><i> {proc_lib,init_p_do_apply,3}]},</i></div>
<div><i> {gen_server,call,</i></div><div><i> [rabbit_mochiweb_registry,</i></div><div><i> {add,rabbit_mgmt_redirect,</i></div><div><i> [{port,55672},{ignore_in_use,true}],</i></div>
<div><i> #Fun<rabbit_mochiweb.1.105980535>,</i></div><div><i> #Fun<rabbit_mochiweb.0.34137573>,</i></div><div><i> {[],"Redirect to port 15672"}},</i></div>
<div><i> infinity]}}}}}</i></div><div><i> type: temporary</i></div></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Dec 20, 2013 at 4:22 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hmm. I cannot figure out how that could come to pass - it looks like there is a broken listener already there.<br>
<br>
You might be able to fix this by invoking<br>
<br>
$ rabbitmqctl -n rabbit eval 'rabbit_web_dispatch_sup:stop_<u></u>listener([{port,55672},{<u></u>ignore_in_use,true}]).'<br>
<br>
but if that doesn't work I think you'd have to restart the node I'm afraid.<br>
<br>
Either way, I'd be very interested to see the logs from around when you tried to restart the management application.<br>
<br>
Cheers, Simon<div class="im"><br>
<br>
On 19/12/13 19:38, shridharan muthu wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Hello Simon,<br>
<br>
I tried restarting rabbitmq management and I got the following<br>
error. I am not able to use port 15672 at all.<br>
<br></div>
/sudo rabbitmqctl -n rabbit eval<br>
'application:stop(rabbitmq_<u></u>management),application:start(<u></u>rabbitmq_management).'/<br>
/{error,/<br>
/ {bad_return,/<br>
/ {{rabbit_mgmt_app,start,[<u></u>normal,[]]},/<br>
/ {'EXIT',/<br>
/ {{{case_clause,/<br>
/ {error,/<br>
/ {no_record_for_listener,/<br>
/ [{port,55672},{ignore_in_use,<u></u>true}]}}},/<br>
/ [{rabbit_mochiweb_registry,<u></u>handle_call,3},/<br>
/ {gen_server,handle_msg,5},/<br>
/ {proc_lib,init_p_do_apply,3}]}<u></u>,/<br>
/ {gen_server,call,/<br>
/ [rabbit_mochiweb_registry,/<br>
/ {add,rabbit_mgmt_redirect,/<br>
/ [{port,55672},{ignore_in_use,<u></u>true}],/<br>
/ #Fun<rabbit_mochiweb.1.<u></u>105980535>,/<br>
/ #Fun<rabbit_mochiweb.0.<u></u>34137573>,/<br>
/ {[],"Redirect to port 15672"}},/<br>
/ infinity]}}}}}}/<br>
/...done./<br>
/<div class="im"><br>
/<br>
So, Is there anyway bring it back up without restarting the app?<br>
<br>
Shri<br>
<br>
<br>
On Thu, Dec 19, 2013 at 9:27 AM, Simon MacMullen <<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a><br></div><div class="im">
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>> wrote:<br>
<br>
On 19/12/13 16:35, shridharan muthu wrote:<br>
<br>
You are spot on Simon. We are using 3.0.2 version in our<br>
cluster. In this case, restarting node will not help I guess. Can I<br>
delete these blocked/blocking connections from Management UI?<br>
<br>
<br>
The only way to do that is to restart the management stats database:<br>
<br>
# rabbitmqctl -n <node with the stats db> eval<br></div>
'application:stop(rabbitmq___<u></u>management),application:start(<u></u>__rabbitmq_management).'<div><div class="h5"><br>
<br>
<br>
Could it be cause of the blocked connections? I will<br>
check the<br>
memory usage after cleaning up the connections.<br>
<br>
<br>
I doubt it, those connections don't really exist except as rows in<br>
the management stats db. But note that we have fixed a few memory<br>
leaks since 3.0.2.<br>
<br>
/You could use a higher priority "nodes" policy to move the<br>
master<br>
<br>
to node 2, then delete it again (albeit the queue would be<br>
unmirrored while this was taking place). Oh, but you need<br>
to be on<br>
at least RabbitMQ 3.1.x for that to work, 3.0.x does not let a<br>
mirroring policy move the master./<br>
<br>
<br>
Ah! But that would leave with tightly coupling queues to nodes.<br>
<br>
<br>
Only until you delete the "nodes" policy - you can use that to move<br>
the master, but then revert to an "all" policy; at which point the<br>
master will stay where it is.<br>
<br>
/OTOH if the queues are mirrored to all nodes anyway, why<br>
worry? A<br>
<br>
slave takes about as many resources as a master anyway, so<br>
all your<br>
masters being on one node and all the slaves on the other<br>
should be<br>
no big deal.<br>
/<br>
<br>
//<br>
<br>
My understanding is that all operations (publish, consume)<br>
on a<br>
queue will be forwarded from slaves to the master (where it will be<br>
executed and broadcasted to all mirrors) and all the messages in the<br>
mirror will be used only when the master goes down.<br>
<br>
<br>
Yes.<br>
<br>
<br>
This sounds like<br>
slaves are just forwarding messages all the time (1 hop penalty) and<br>
generates more traffic between slaves & the master.<br>
<br>
<br>
Sure, but how will distributing the masters around the cluster help<br>
that? Or are you saying your clients know that some queues are on<br>
node 2, and expect to connect to that node to use that queue? If so<br>
that makes sense...<br>
<br>
<br>
Cheers, Simon<br>
<br>
--<br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
<br>
<br>
</div></div></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</div></div></blockquote></div><br></div>