[rabbitmq-discuss] RabbitMQ very slow (or never) shuts down
Matt Pietrek
mpietrek at skytap.com
Thu Sep 13 01:00:32 BST 2012
Hold on, there's more output that appeared after I sent the original email.
This instance took 14 minutes to shut down.
The additional rabbit at mq1-sasl.log file looks like this:
=CRASH REPORT==== 12-Sep-2012::16:51:56 ===
crasher:
initial call: amqp_main_reader:init/1
pid: <0.316.0>
registered_name: []
exception exit: {socket_error,etimedout}
in function gen_server:terminate/6 (gen_server.erl, line 747)
ancestors: [<0.314.0>,<0.310.0>,amqp_sup,<0.245.0>]
messages: []
links: [<0.314.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 2142252
neighbours:
=CRASH REPORT==== 12-Sep-2012::16:51:56 ===
crasher:
initial call: amqp_gen_connection:init/1
pid: <0.311.0>
registered_name: []
exception exit: {socket_error,etimedout}
in function gen_server:terminate/6 (gen_server.erl, line 747)
ancestors: [<0.310.0>,amqp_sup,<0.245.0>]
messages: []
links: [<0.310.0>,#Port<0.5488>]
dictionary: []
trap_exit: false
status: running
heap_size: 1597
stack_size: 24
reductions: 805
neighbours:
=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
Supervisor: {<0.314.0>,amqp_connection_type_sup}
Context: child_terminated
Reason: {socket_error,etimedout}
Offender: [{pid,<0.316.0>},
{name,main_reader},
{mfa,
{amqp_main_reader,start_link,
[#Port<0.5488>,<0.311.0>,<0.313.0>,
{method,rabbit_framing_amqp_0_9_1}]}},
{restart_type,transient},
{shutdown,4294967295},
{child_type,worker}]
=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
Supervisor: {<0.314.0>,amqp_connection_type_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [{pid,<0.316.0>},
{name,main_reader},
{mfa,
{amqp_main_reader,start_link,
[#Port<0.5488>,<0.311.0>,<0.313.0>,
{method,rabbit_framing_amqp_0_9_1}]}},
{restart_type,transient},
{shutdown,4294967295},
{child_type,worker}]
=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
Supervisor: {<0.310.0>,amqp_connection_sup}
Context: child_terminated
Reason: {socket_error,etimedout}
Offender: [{pid,<0.311.0>},
{name,connection},
{mfa,
{amqp_gen_connection,start_link,
[amqp_network_connection,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"slg1matt1.mgmt.xyz.com",
5672,0,0,0,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
#Fun<amqp_connection_sup.0.39273983>,
#Fun<amqp_connection_sup.2.54430129>,[]]}},
{restart_type,intrinsic},
{shutdown,brutal_kill},
{child_type,worker}]
=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
Supervisor: {<0.310.0>,amqp_connection_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [{pid,<0.311.0>},
{name,connection},
{mfa,
{amqp_gen_connection,start_link,
[amqp_network_connection,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"slg1matt1.mgmt.xyz.com",
5672,0,0,0,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
#Fun<amqp_connection_sup.0.39273983>,
#Fun<amqp_connection_sup.2.54430129>,[]]}},
{restart_type,intrinsic},
{shutdown,brutal_kill},
{child_type,worker}]
=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
Supervisor: {<0.310.0>,amqp_connection_sup}
Context: shutdown_error
Reason: shutdown
Offender: [{pid,<0.314.0>},
{name,connection_type_sup},
{mfa,
{amqp_connection_type_sup,start_link_network,
[#Port<0.5488>,<0.311.0>,<0.313.0>]}},
{restart_type,transient},
{shutdown,infinity},
{child_type,supervisor}]
=CRASH REPORT==== 12-Sep-2012::16:51:56 ===
crasher:
initial call: gen:init_it/6
pid: <0.200.0>
registered_name: []
exception exit: {undef,
[{rabbit_federation_old_status,report,
[{upstream,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"slg1matt1.mgmt.xyz.com",5672,0,
0,0,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
<<"skytap">>,1000,1,1,none,none,none,"upstream-slg"},
{resource,<<"/">>,exchange,<<"skytap">>},
{shutdown,
{gen_server,call,
[<0.319.0>,
{call,
{'exchange.unbind',0,
<<"federation: skytap ->
rabbit at mq1.matt.internal.xyz.com B">>,
<<"skytap">>,
<<"ConfigurationManager.us-east.default">>,false,
[]},
none,<0.200.0>},
infinity]}}],
[]},
{rabbit_federation_link,terminate,2,[]},
{gen_server2,terminate,3,[]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]}
in function gen_server2:terminate/3
ancestors: [<0.199.0>,<0.196.0>,rabbit_federation_link_sup_sup,
rabbit_federation_sup,rabbit_sup,<0.145.0>]
messages: [{'$gen_cast',
{enqueue,23,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"ConfigurationManager.us-west.default">>,
{resource,<<"/">>,queue,
<<"ConfigurationManager">>},
[]}]}}},
{'$gen_cast',
{enqueue,24,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"ConfigurationManager.us-west.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,25,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"charon.us-east.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,26,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"charon.us-west.default">>,
{resource,<<"/">>,queue,<<"charon">>},
[]}]}}},
{'$gen_cast',
{enqueue,27,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"charon.us-west.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,28,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"cmcmd.us-east.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,29,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"cmcmd.us-west.default">>,
{resource,<<"/">>,queue,<<"cmcmd">>},
[]}]}}},
{'$gen_cast',
{enqueue,30,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"cmcmd.us-west.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,31,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"system_test.us-east.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,32,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"system_test.us-west.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,33,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"system_test.us-west.default">>,
{resource,<<"/">>,queue,<<"system_test">>},
[]}]}}},
{'$gen_cast',
{enqueue,34,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"unit_test.us-east.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,35,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"unit_test.us-west.default">>,
{resource,<<"/">>,exchange,
<<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
[]}]}}},
{'$gen_cast',
{enqueue,36,
{remove_bindings,
[{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"unit_test.us-west.default">>,
{resource,<<"/">>,queue,<<"unit_test">>},
[]}]}}},
{'$gen_cast',
{enqueue,37,
{add_binding,
{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"charon.us-west.default">>,
{resource,<<"/">>,queue,<<"charon">>},
[]}}}},
{'$gen_cast',
{enqueue,38,
{add_binding,
{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"ConfigurationManager.us-west.default">>,
{resource,<<"/">>,queue,
<<"ConfigurationManager">>},
[]}}}},
{'$gen_cast',
{enqueue,39,
{add_binding,
{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"unit_test.us-west.default">>,
{resource,<<"/">>,queue,<<"unit_test">>},
[]}}}},
{'$gen_cast',
{enqueue,40,
{add_binding,
{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"system_test.us-west.default">>,
{resource,<<"/">>,queue,<<"system_test">>},
[]}}}},
{'$gen_cast',
{enqueue,41,
{add_binding,
{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"cmcmd.us-west.default">>,
{resource,<<"/">>,queue,<<"cmcmd">>},
[]}}}},
{'DOWN',#Ref<0.0.0.1656>,process,<0.306.0>,shutdown},
{'EXIT',<0.199.0>,shutdown},
{'DOWN',#Ref<0.0.0.1688>,process,<0.319.0>,shutdown}]
links: []
dictionary: []
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 1851087
neighbours:
=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
Supervisor: {<0.199.0>,
rabbit_federation_link_sup}
Context: shutdown_error
Reason: {undef,
[{rabbit_federation_old_status,report,
[{upstream,
{amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,
"slg1matt1.mgmt.xyz.com",5672,0,0,0,
infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
<<"skytap">>,1000,1,1,none,none,none,"upstream-slg"},
{resource,<<"/">>,exchange,<<"skytap">>},
{shutdown,
{gen_server,call,
[<0.319.0>,
{call,
{'exchange.unbind',0,
<<"federation: skytap ->
rabbit at mq1.matt.internal.xyz.com B">>,
<<"skytap">>,
<<"ConfigurationManager.us-east.default">>,false,[]},
none,<0.200.0>},
infinity]}}],
[]},
{rabbit_federation_link,terminate,2,[]},
{gen_server2,terminate,3,[]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]}
Offender: [{pid,<0.200.0>},
{name,
{upstream,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"slg1matt1.mgmt.xyz.com",
5672,0,0,0,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
<<"skytap">>,1000,1,1,none,none,none,
"upstream-slg"}},
{mfa,
{rabbit_federation_link,start_link,
[{{upstream,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,
"slg1matt1.mgmt.xyz.com",5672,
0,0,0,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
<<"skytap">>,1000,1,1,none,none,none,
"upstream-slg"},
{resource,<<"/">>,exchange,<<"skytap">>}}]}},
{restart_type,{permanent,1}},
{shutdown,4294967295},
{child_type,worker}]
On Wed, Sep 12, 2012 at 4:51 PM, Matt Pietrek <mpietrek at skytap.com> wrote:
> I'm experience a problem where "rabbitmqctl stop" takes a very long time
> (5+ minutes, or never) shuts down.
>
> This is RabbitMQ 2.8.6 and Erlang R15B01 on Ubuntu 10.04.
>
> The broker is running with the Federation plug-in in use, FWIW.
>
> The last little bits of console output are:
>
> --------
> Mnesia(rabbit at mq1): write performed by {tid,307,<0.151.0>} on record:
> {rabbit_durable_route,
> {binding,
> {resource,<<"/">>,exchange,<<"skytap">>},
> <<"cmcmd.us-west.default">>,
> {resource,<<"/">>,queue,<<"cmcmd">>},
> []},
> const}
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> {needs_dump,40}
> Mnesia(rabbit at mq1): delete_object performed by {dirty,<0.237.0>} on
> record:
> {rabbit_listener,rabbit at mq1,amqp,"mq1.blah.com",
> {0,0,0,0},
> 5672}
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> already_dumped
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> already_dumped
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> already_dumped
> --------
>
> The rabbit at mq1.log file:
>
> -------
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
> Stopping Rabbit
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
> application: rabbitmq_management
> exited: stopped
> type: permanent
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
> application: rabbitmq_federation
> exited: stopped
> type: permanent
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
> application: rabbitmq_management_agent
> exited: stopped
> type: permanent
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
> stopped TCP Listener on 0.0.0.0:5672
>
> =WARNING REPORT==== 12-Sep-2012::16:38:14 ===
> Connection (<0.299.0>) closing: internal error in channel (<0.306.0>):
> shutdown
> -------
>
> And the rabbitm at mq1-sasl log:
>
> --------
> =SUPERVISOR REPORT==== 12-Sep-2012::16:38:14 ===
> Supervisor: {<0.304.0>,amqp_channel_sup}
> Context: child_terminated
> Reason: shutdown
> Offender: [{pid,<0.306.0>},
> {name,channel},
> {mfa,{amqp_channel,start_link,
> [direct,<0.299.0>,1,<0.305.0>,
> #Fun<amqp_channel_sup.0.24953263>]}},
> {restart_type,intrinsic},
> {shutdown,4294967295},
> {child_type,worker}]
>
>
> =SUPERVISOR REPORT==== 12-Sep-2012::16:38:14 ===
> Supervisor: {<0.304.0>,amqp_channel_sup}
> Context: shutdown
> Reason: reached_max_restart_intensity
> Offender: [{pid,<0.306.0>},
> {name,channel},
> {mfa,{amqp_channel,start_link,
> [direct,<0.299.0>,1,<0.305.0>,
> #Fun<amqp_channel_sup.0.24953263>]}},
> {restart_type,intrinsic},
> {shutdown,4294967295},
> {child_type,worker}]
> --------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120912/99994bd7/attachment.htm>
More information about the rabbitmq-discuss
mailing list