Hold on, there's more output that appeared after I sent the original email. This instance took 14 minutes to shut down.<br><br>The additional rabbit@mq1-sasl.log file looks like this:<br><br>=CRASH REPORT==== 12-Sep-2012::16:51:56 ===<br>
crasher:<br> initial call: amqp_main_reader:init/1<br> pid: <0.316.0><br> registered_name: []<br> exception exit: {socket_error,etimedout}<br> in function gen_server:terminate/6 (gen_server.erl, line 747)<br>
ancestors: [<0.314.0>,<0.310.0>,amqp_sup,<0.245.0>]<br> messages: []<br> links: [<0.314.0>]<br> dictionary: []<br> trap_exit: false<br> status: running<br> heap_size: 987<br> stack_size: 24<br>
reductions: 2142252<br> neighbours:<br><br>=CRASH REPORT==== 12-Sep-2012::16:51:56 ===<br> crasher:<br> initial call: amqp_gen_connection:init/1<br> pid: <0.311.0><br> registered_name: []<br> exception exit: {socket_error,etimedout}<br>
in function gen_server:terminate/6 (gen_server.erl, line 747)<br> ancestors: [<0.310.0>,amqp_sup,<0.245.0>]<br> messages: []<br> links: [<0.310.0>,#Port<0.5488>]<br> dictionary: []<br>
trap_exit: false<br> status: running<br> heap_size: 1597<br> stack_size: 24<br> reductions: 805<br> neighbours:<br><br>=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===<br> Supervisor: {<0.314.0>,amqp_connection_type_sup}<br>
Context: child_terminated<br> Reason: {socket_error,etimedout}<br> Offender: [{pid,<0.316.0>},<br> {name,main_reader},<br> {mfa,<br> {amqp_main_reader,start_link,<br>
[#Port<0.5488>,<0.311.0>,<0.313.0>,<br> {method,rabbit_framing_amqp_0_9_1}]}},<br> {restart_type,transient},<br> {shutdown,4294967295},<br>
{child_type,worker}]<br><br><br>=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===<br> Supervisor: {<0.314.0>,amqp_connection_type_sup}<br> Context: shutdown<br> Reason: reached_max_restart_intensity<br>
Offender: [{pid,<0.316.0>},<br> {name,main_reader},<br> {mfa,<br> {amqp_main_reader,start_link,<br> [#Port<0.5488>,<0.311.0>,<0.313.0>,<br>
{method,rabbit_framing_amqp_0_9_1}]}},<br> {restart_type,transient},<br> {shutdown,4294967295},<br> {child_type,worker}]<br><br><br>=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===<br>
Supervisor: {<0.310.0>,amqp_connection_sup}<br> Context: child_terminated<br> Reason: {socket_error,etimedout}<br> Offender: [{pid,<0.311.0>},<br> {name,connection},<br>
{mfa,<br> {amqp_gen_connection,start_link,<br> [amqp_network_connection,<br> {amqp_params_network,<<"guest">>,<<"guest">>,<br>
<<"/">>,"<a href="http://slg1matt1.mgmt.xyz.com">slg1matt1.mgmt.xyz.com</a>",<br> 5672,0,0,0,infinity,none,<br> [#Fun<amqp_auth_mechanisms.plain.3>,<br>
#Fun<amqp_auth_mechanisms.amqplain.3>],<br> [],[]},<br> #Fun<amqp_connection_sup.0.39273983>,<br> #Fun<amqp_connection_sup.2.54430129>,[]]}},<br>
{restart_type,intrinsic},<br> {shutdown,brutal_kill},<br> {child_type,worker}]<br><br><br>=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===<br> Supervisor: {<0.310.0>,amqp_connection_sup}<br>
Context: shutdown<br> Reason: reached_max_restart_intensity<br> Offender: [{pid,<0.311.0>},<br> {name,connection},<br> {mfa,<br> {amqp_gen_connection,start_link,<br>
[amqp_network_connection,<br> {amqp_params_network,<<"guest">>,<<"guest">>,<br> <<"/">>,"<a href="http://slg1matt1.mgmt.xyz.com">slg1matt1.mgmt.xyz.com</a>",<br>
5672,0,0,0,infinity,none,<br> [#Fun<amqp_auth_mechanisms.plain.3>,<br> #Fun<amqp_auth_mechanisms.amqplain.3>],<br> [],[]},<br>
#Fun<amqp_connection_sup.0.39273983>,<br> #Fun<amqp_connection_sup.2.54430129>,[]]}},<br> {restart_type,intrinsic},<br> {shutdown,brutal_kill},<br>
{child_type,worker}]<br><br><br>=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===<br> Supervisor: {<0.310.0>,amqp_connection_sup}<br> Context: shutdown_error<br> Reason: shutdown<br>
Offender: [{pid,<0.314.0>},<br> {name,connection_type_sup},<br> {mfa,<br> {amqp_connection_type_sup,start_link_network,<br> [#Port<0.5488>,<0.311.0>,<0.313.0>]}},<br>
{restart_type,transient},<br> {shutdown,infinity},<br> {child_type,supervisor}]<br><br><br>=CRASH REPORT==== 12-Sep-2012::16:51:56 ===<br> crasher:<br> initial call: gen:init_it/6<br>
pid: <0.200.0><br> registered_name: []<br> exception exit: {undef,<br> [{rabbit_federation_old_status,report,<br> [{upstream,<br> {amqp_params_network,<<"guest">>,<<"guest">>,<br>
<<"/">>,"<a href="http://slg1matt1.mgmt.xyz.com">slg1matt1.mgmt.xyz.com</a>",5672,0,<br> 0,0,infinity,none,<br> [#Fun<amqp_auth_mechanisms.plain.3>,<br>
#Fun<amqp_auth_mechanisms.amqplain.3>],<br> [],[]},<br> <<"skytap">>,1000,1,1,none,none,none,"upstream-slg"},<br>
{resource,<<"/">>,exchange,<<"skytap">>},<br> {shutdown,<br> {gen_server,call,<br> [<0.319.0>,<br>
{call,<br> {'exchange.unbind',0,<br> <<"federation: skytap -> <a href="mailto:rabbit@mq1.matt.internal.xyz.com">rabbit@mq1.matt.internal.xyz.com</a> B">>,<br>
<<"skytap">>,<br> <<"ConfigurationManager.us-east.default">>,false,<br> []},<br> none,<0.200.0>},<br>
infinity]}}],<br> []},<br> {rabbit_federation_link,terminate,2,[]},<br> {gen_server2,terminate,3,[]},<br> {proc_lib,init_p_do_apply,3,<br>
[{file,"proc_lib.erl"},{line,227}]}]}<br> in function gen_server2:terminate/3 <br> ancestors: [<0.199.0>,<0.196.0>,rabbit_federation_link_sup_sup,<br> rabbit_federation_sup,rabbit_sup,<0.145.0>]<br>
messages: [{'$gen_cast',<br> {enqueue,23,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"ConfigurationManager.us-west.default">>,<br> {resource,<<"/">>,queue,<br> <<"ConfigurationManager">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,24,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"ConfigurationManager.us-west.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,25,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"charon.us-east.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,26,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"charon.us-west.default">>,<br> {resource,<<"/">>,queue,<<"charon">>},<br> []}]}}},<br>
{'$gen_cast',<br> {enqueue,27,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"charon.us-west.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,28,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"cmcmd.us-east.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,29,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"cmcmd.us-west.default">>,<br> {resource,<<"/">>,queue,<<"cmcmd">>},<br> []}]}}},<br>
{'$gen_cast',<br> {enqueue,30,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"cmcmd.us-west.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,31,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"system_test.us-east.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,32,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"system_test.us-west.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,33,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"system_test.us-west.default">>,<br> {resource,<<"/">>,queue,<<"system_test">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,34,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"unit_test.us-east.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,35,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"unit_test.us-west.default">>,<br> {resource,<<"/">>,exchange,<br> <<"federation: skytap -> <a href="mailto:rabbit@slg1matt1.mgmt.xyz.com">rabbit@slg1matt1.mgmt.xyz.com</a> B">>},<br>
[]}]}}},<br> {'$gen_cast',<br> {enqueue,36,<br> {remove_bindings,<br> [{binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"unit_test.us-west.default">>,<br> {resource,<<"/">>,queue,<<"unit_test">>},<br> []}]}}},<br>
{'$gen_cast',<br> {enqueue,37,<br> {add_binding,<br> {binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"charon.us-west.default">>,<br> {resource,<<"/">>,queue,<<"charon">>},<br> []}}}},<br>
{'$gen_cast',<br> {enqueue,38,<br> {add_binding,<br> {binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"ConfigurationManager.us-west.default">>,<br> {resource,<<"/">>,queue,<br> <<"ConfigurationManager">>},<br>
[]}}}},<br> {'$gen_cast',<br> {enqueue,39,<br> {add_binding,<br> {binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"unit_test.us-west.default">>,<br> {resource,<<"/">>,queue,<<"unit_test">>},<br> []}}}},<br>
{'$gen_cast',<br> {enqueue,40,<br> {add_binding,<br> {binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"system_test.us-west.default">>,<br> {resource,<<"/">>,queue,<<"system_test">>},<br> []}}}},<br>
{'$gen_cast',<br> {enqueue,41,<br> {add_binding,<br> {binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"cmcmd.us-west.default">>,<br> {resource,<<"/">>,queue,<<"cmcmd">>},<br> []}}}},<br>
{'DOWN',#Ref<0.0.0.1656>,process,<0.306.0>,shutdown},<br> {'EXIT',<0.199.0>,shutdown},<br> {'DOWN',#Ref<0.0.0.1688>,process,<0.319.0>,shutdown}]<br>
links: []<br> dictionary: []<br> trap_exit: true<br> status: running<br> heap_size: 4181<br> stack_size: 24<br> reductions: 1851087<br> neighbours:<br><br>=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===<br>
Supervisor: {<0.199.0>,<br> rabbit_federation_link_sup}<br> Context: shutdown_error<br> Reason: {undef,<br> [{rabbit_federation_old_status,report,<br>
[{upstream,<br> {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,<br> "<a href="http://slg1matt1.mgmt.xyz.com">slg1matt1.mgmt.xyz.com</a>",5672,0,0,0,<br>
infinity,none,<br> [#Fun<amqp_auth_mechanisms.plain.3>,<br> #Fun<amqp_auth_mechanisms.amqplain.3>],<br> [],[]},<br> <<"skytap">>,1000,1,1,none,none,none,"upstream-slg"},<br>
{resource,<<"/">>,exchange,<<"skytap">>},<br> {shutdown,<br> {gen_server,call,<br> [<0.319.0>,<br>
{call,<br> {'exchange.unbind',0,<br> <<"federation: skytap -> <a href="mailto:rabbit@mq1.matt.internal.xyz.com">rabbit@mq1.matt.internal.xyz.com</a> B">>,<br>
<<"skytap">>,<br> <<"ConfigurationManager.us-east.default">>,false,[]},<br> none,<0.200.0>},<br>
infinity]}}],<br> []},<br> {rabbit_federation_link,terminate,2,[]},<br> {gen_server2,terminate,3,[]},<br> {proc_lib,init_p_do_apply,3,<br>
[{file,"proc_lib.erl"},{line,227}]}]}<br> Offender: [{pid,<0.200.0>},<br> {name,<br> {upstream,<br> {amqp_params_network,<<"guest">>,<<"guest">>,<br>
<<"/">>,"<a href="http://slg1matt1.mgmt.xyz.com">slg1matt1.mgmt.xyz.com</a>",<br> 5672,0,0,0,infinity,none,<br> [#Fun<amqp_auth_mechanisms.plain.3>,<br>
#Fun<amqp_auth_mechanisms.amqplain.3>],<br> [],[]},<br> <<"skytap">>,1000,1,1,none,none,none,<br> "upstream-slg"}},<br>
{mfa,<br> {rabbit_federation_link,start_link,<br> [{{upstream,<br> {amqp_params_network,<<"guest">>,<<"guest">>,<br>
<<"/">>,<br> "<a href="http://slg1matt1.mgmt.xyz.com">slg1matt1.mgmt.xyz.com</a>",5672,<br> 0,0,0,infinity,none,<br>
[#Fun<amqp_auth_mechanisms.plain.3>,<br> #Fun<amqp_auth_mechanisms.amqplain.3>],<br> [],[]},<br> <<"skytap">>,1000,1,1,none,none,none,<br>
"upstream-slg"},<br> {resource,<<"/">>,exchange,<<"skytap">>}}]}},<br> {restart_type,{permanent,1}},<br>
{shutdown,4294967295},<br> {child_type,worker}]<br><br><div class="gmail_quote">On Wed, Sep 12, 2012 at 4:51 PM, Matt Pietrek <span dir="ltr"><<a href="mailto:mpietrek@skytap.com" target="_blank">mpietrek@skytap.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm experience a problem where "rabbitmqctl stop" takes a very long time (5+ minutes, or never) shuts down.<br>
<br>This is RabbitMQ 2.8.6 and Erlang R15B01 on Ubuntu 10.04.<br><br>The broker is running with the Federation plug-in in use, FWIW.<br>
<br>The last little bits of console output are:<br><br>--------<br>Mnesia(rabbit@mq1): write performed by {tid,307,<0.151.0>} on record:<br> {rabbit_durable_route,<br> {binding,<br> {resource,<<"/">>,exchange,<<"skytap">>},<br>
<<"cmcmd.us-west.default">>,<br> {resource,<<"/">>,queue,<<"cmcmd">>},<br> []},<br> const}<br>Mnesia(rabbit@mq1): Transaction log dump initiated by time_threshold: {needs_dump,40}<br>
Mnesia(rabbit@mq1): delete_object performed by {dirty,<0.237.0>} on record:<br> {rabbit_listener,rabbit@mq1,amqp,"<a href="http://mq1.blah.com" target="_blank">mq1.blah.com</a>",<br> {0,0,0,0},<br>
5672}<br>Mnesia(rabbit@mq1): Transaction log dump initiated by time_threshold: already_dumped<br>Mnesia(rabbit@mq1): Transaction log dump initiated by time_threshold: already_dumped<br>Mnesia(rabbit@mq1): Transaction log dump initiated by time_threshold: already_dumped<br>
--------<br><br>The rabbit@mq1.log file:<br><br>-------<br>=INFO REPORT==== 12-Sep-2012::16:38:14 ===<br>Stopping Rabbit<br><br>=INFO REPORT==== 12-Sep-2012::16:38:14 ===<br> application: rabbitmq_management<br> exited: stopped<br>
type: permanent<br><br>=INFO REPORT==== 12-Sep-2012::16:38:14 ===<br> application: rabbitmq_federation<br> exited: stopped<br> type: permanent<br><br>=INFO REPORT==== 12-Sep-2012::16:38:14 ===<br> application: rabbitmq_management_agent<br>
exited: stopped<br> type: permanent<br><br>=INFO REPORT==== 12-Sep-2012::16:38:14 ===<br>stopped TCP Listener on <a href="http://0.0.0.0:5672" target="_blank">0.0.0.0:5672</a><br><br>=WARNING REPORT==== 12-Sep-2012::16:38:14 ===<br>
Connection (<0.299.0>) closing: internal error in channel (<0.306.0>): shutdown<br>-------<br><br>And the rabbitm@mq1-sasl log:<br><br>--------<br>=SUPERVISOR REPORT==== 12-Sep-2012::16:38:14 ===<br> Supervisor: {<0.304.0>,amqp_channel_sup}<br>
Context: child_terminated<br> Reason: shutdown<br> Offender: [{pid,<0.306.0>},<br> {name,channel},<br> {mfa,{amqp_channel,start_link,<br> [direct,<0.299.0>,1,<0.305.0>,<br>
#Fun<amqp_channel_sup.0.24953263>]}},<br> {restart_type,intrinsic},<br> {shutdown,4294967295},<br> {child_type,worker}]<br><br>
<br>=SUPERVISOR REPORT==== 12-Sep-2012::16:38:14 ===<br> Supervisor: {<0.304.0>,amqp_channel_sup}<br> Context: shutdown<br> Reason: reached_max_restart_intensity<br> Offender: [{pid,<0.306.0>},<br>
{name,channel},<br> {mfa,{amqp_channel,start_link,<br> [direct,<0.299.0>,1,<0.305.0>,<br> #Fun<amqp_channel_sup.0.24953263>]}},<br>
{restart_type,intrinsic},<br> {shutdown,4294967295},<br> {child_type,worker}]<br>--------<br>
</blockquote></div><br>