[rabbitmq-discuss] RabbitMQ very slow (or never) shuts down

Matt Pietrek mpietrek at skytap.com
Thu Sep 13 01:00:32 BST 2012


Hold on, there's more output that appeared after I sent the original email.
This instance took 14 minutes to shut down.

The additional rabbit at mq1-sasl.log file looks like this:

=CRASH REPORT==== 12-Sep-2012::16:51:56 ===
  crasher:
    initial call: amqp_main_reader:init/1
    pid: <0.316.0>
    registered_name: []
    exception exit: {socket_error,etimedout}
      in function  gen_server:terminate/6 (gen_server.erl, line 747)
    ancestors: [<0.314.0>,<0.310.0>,amqp_sup,<0.245.0>]
    messages: []
    links: [<0.314.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 2142252
  neighbours:

=CRASH REPORT==== 12-Sep-2012::16:51:56 ===
  crasher:
    initial call: amqp_gen_connection:init/1
    pid: <0.311.0>
    registered_name: []
    exception exit: {socket_error,etimedout}
      in function  gen_server:terminate/6 (gen_server.erl, line 747)
    ancestors: [<0.310.0>,amqp_sup,<0.245.0>]
    messages: []
    links: [<0.310.0>,#Port<0.5488>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 805
  neighbours:

=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
     Supervisor: {<0.314.0>,amqp_connection_type_sup}
     Context:    child_terminated
     Reason:     {socket_error,etimedout}
     Offender:   [{pid,<0.316.0>},
                  {name,main_reader},
                  {mfa,
                      {amqp_main_reader,start_link,
                          [#Port<0.5488>,<0.311.0>,<0.313.0>,
                           {method,rabbit_framing_amqp_0_9_1}]}},
                  {restart_type,transient},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
     Supervisor: {<0.314.0>,amqp_connection_type_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.316.0>},
                  {name,main_reader},
                  {mfa,
                      {amqp_main_reader,start_link,
                          [#Port<0.5488>,<0.311.0>,<0.313.0>,
                           {method,rabbit_framing_amqp_0_9_1}]}},
                  {restart_type,transient},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
     Supervisor: {<0.310.0>,amqp_connection_sup}
     Context:    child_terminated
     Reason:     {socket_error,etimedout}
     Offender:   [{pid,<0.311.0>},
                  {name,connection},
                  {mfa,
                      {amqp_gen_connection,start_link,
                          [amqp_network_connection,
                           {amqp_params_network,<<"guest">>,<<"guest">>,
                               <<"/">>,"slg1matt1.mgmt.xyz.com",
                               5672,0,0,0,infinity,none,
                               [#Fun<amqp_auth_mechanisms.plain.3>,
                                #Fun<amqp_auth_mechanisms.amqplain.3>],
                               [],[]},
                           #Fun<amqp_connection_sup.0.39273983>,
                           #Fun<amqp_connection_sup.2.54430129>,[]]}},
                  {restart_type,intrinsic},
                  {shutdown,brutal_kill},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
     Supervisor: {<0.310.0>,amqp_connection_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.311.0>},
                  {name,connection},
                  {mfa,
                      {amqp_gen_connection,start_link,
                          [amqp_network_connection,
                           {amqp_params_network,<<"guest">>,<<"guest">>,
                               <<"/">>,"slg1matt1.mgmt.xyz.com",
                               5672,0,0,0,infinity,none,
                               [#Fun<amqp_auth_mechanisms.plain.3>,
                                #Fun<amqp_auth_mechanisms.amqplain.3>],
                               [],[]},
                           #Fun<amqp_connection_sup.0.39273983>,
                           #Fun<amqp_connection_sup.2.54430129>,[]]}},
                  {restart_type,intrinsic},
                  {shutdown,brutal_kill},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
     Supervisor: {<0.310.0>,amqp_connection_sup}
     Context:    shutdown_error
     Reason:     shutdown
     Offender:   [{pid,<0.314.0>},
                  {name,connection_type_sup},
                  {mfa,
                      {amqp_connection_type_sup,start_link_network,
                          [#Port<0.5488>,<0.311.0>,<0.313.0>]}},
                  {restart_type,transient},
                  {shutdown,infinity},
                  {child_type,supervisor}]


=CRASH REPORT==== 12-Sep-2012::16:51:56 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.200.0>
    registered_name: []
    exception exit: {undef,
                     [{rabbit_federation_old_status,report,
                       [{upstream,
                         {amqp_params_network,<<"guest">>,<<"guest">>,
                          <<"/">>,"slg1matt1.mgmt.xyz.com",5672,0,
                          0,0,infinity,none,
                          [#Fun<amqp_auth_mechanisms.plain.3>,
                           #Fun<amqp_auth_mechanisms.amqplain.3>],
                          [],[]},

<<"skytap">>,1000,1,1,none,none,none,"upstream-slg"},
                        {resource,<<"/">>,exchange,<<"skytap">>},
                        {shutdown,
                         {gen_server,call,
                          [<0.319.0>,
                           {call,
                            {'exchange.unbind',0,
                             <<"federation: skytap ->
rabbit at mq1.matt.internal.xyz.com B">>,
                             <<"skytap">>,

<<"ConfigurationManager.us-east.default">>,false,
                             []},
                            none,<0.200.0>},
                           infinity]}}],
                       []},
                      {rabbit_federation_link,terminate,2,[]},
                      {gen_server2,terminate,3,[]},
                      {proc_lib,init_p_do_apply,3,
                       [{file,"proc_lib.erl"},{line,227}]}]}
      in function  gen_server2:terminate/3
    ancestors: [<0.199.0>,<0.196.0>,rabbit_federation_link_sup_sup,
                  rabbit_federation_sup,rabbit_sup,<0.145.0>]
    messages: [{'$gen_cast',
                      {enqueue,23,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},

<<"ConfigurationManager.us-west.default">>,
                                   {resource,<<"/">>,queue,
                                       <<"ConfigurationManager">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,24,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},

<<"ConfigurationManager.us-west.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,25,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"charon.us-east.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,26,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"charon.us-west.default">>,
                                   {resource,<<"/">>,queue,<<"charon">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,27,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"charon.us-west.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,28,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"cmcmd.us-east.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,29,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"cmcmd.us-west.default">>,
                                   {resource,<<"/">>,queue,<<"cmcmd">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,30,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"cmcmd.us-west.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,31,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"system_test.us-east.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,32,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"system_test.us-west.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,33,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"system_test.us-west.default">>,

{resource,<<"/">>,queue,<<"system_test">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,34,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"unit_test.us-east.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,35,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"unit_test.us-west.default">>,
                                   {resource,<<"/">>,exchange,
                                       <<"federation: skytap ->
rabbit at slg1matt1.mgmt.xyz.com B">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,36,
                          {remove_bindings,
                              [{binding,
                                   {resource,<<"/">>,exchange,<<"skytap">>},
                                   <<"unit_test.us-west.default">>,
                                   {resource,<<"/">>,queue,<<"unit_test">>},
                                   []}]}}},
                  {'$gen_cast',
                      {enqueue,37,
                          {add_binding,
                              {binding,
                                  {resource,<<"/">>,exchange,<<"skytap">>},
                                  <<"charon.us-west.default">>,
                                  {resource,<<"/">>,queue,<<"charon">>},
                                  []}}}},
                  {'$gen_cast',
                      {enqueue,38,
                          {add_binding,
                              {binding,
                                  {resource,<<"/">>,exchange,<<"skytap">>},

<<"ConfigurationManager.us-west.default">>,
                                  {resource,<<"/">>,queue,
                                      <<"ConfigurationManager">>},
                                  []}}}},
                  {'$gen_cast',
                      {enqueue,39,
                          {add_binding,
                              {binding,
                                  {resource,<<"/">>,exchange,<<"skytap">>},
                                  <<"unit_test.us-west.default">>,
                                  {resource,<<"/">>,queue,<<"unit_test">>},
                                  []}}}},
                  {'$gen_cast',
                      {enqueue,40,
                          {add_binding,
                              {binding,
                                  {resource,<<"/">>,exchange,<<"skytap">>},
                                  <<"system_test.us-west.default">>,

{resource,<<"/">>,queue,<<"system_test">>},
                                  []}}}},
                  {'$gen_cast',
                      {enqueue,41,
                          {add_binding,
                              {binding,
                                  {resource,<<"/">>,exchange,<<"skytap">>},
                                  <<"cmcmd.us-west.default">>,
                                  {resource,<<"/">>,queue,<<"cmcmd">>},
                                  []}}}},
                  {'DOWN',#Ref<0.0.0.1656>,process,<0.306.0>,shutdown},
                  {'EXIT',<0.199.0>,shutdown},
                  {'DOWN',#Ref<0.0.0.1688>,process,<0.319.0>,shutdown}]
    links: []
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 4181
    stack_size: 24
    reductions: 1851087
  neighbours:

=SUPERVISOR REPORT==== 12-Sep-2012::16:51:56 ===
     Supervisor: {<0.199.0>,
                                           rabbit_federation_link_sup}
     Context:    shutdown_error
     Reason:     {undef,
                  [{rabbit_federation_old_status,report,
                    [{upstream,
                      {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,
                       "slg1matt1.mgmt.xyz.com",5672,0,0,0,
                       infinity,none,
                       [#Fun<amqp_auth_mechanisms.plain.3>,
                        #Fun<amqp_auth_mechanisms.amqplain.3>],
                       [],[]},
                      <<"skytap">>,1000,1,1,none,none,none,"upstream-slg"},
                     {resource,<<"/">>,exchange,<<"skytap">>},
                     {shutdown,
                      {gen_server,call,
                       [<0.319.0>,
                        {call,
                         {'exchange.unbind',0,
                          <<"federation: skytap ->
rabbit at mq1.matt.internal.xyz.com B">>,
                          <<"skytap">>,

<<"ConfigurationManager.us-east.default">>,false,[]},
                         none,<0.200.0>},
                        infinity]}}],
                    []},
                   {rabbit_federation_link,terminate,2,[]},
                   {gen_server2,terminate,3,[]},
                   {proc_lib,init_p_do_apply,3,
                    [{file,"proc_lib.erl"},{line,227}]}]}
     Offender:   [{pid,<0.200.0>},
                  {name,
                      {upstream,
                          {amqp_params_network,<<"guest">>,<<"guest">>,
                              <<"/">>,"slg1matt1.mgmt.xyz.com",
                              5672,0,0,0,infinity,none,
                              [#Fun<amqp_auth_mechanisms.plain.3>,
                               #Fun<amqp_auth_mechanisms.amqplain.3>],
                              [],[]},
                          <<"skytap">>,1000,1,1,none,none,none,
                          "upstream-slg"}},
                  {mfa,
                      {rabbit_federation_link,start_link,
                          [{{upstream,

{amqp_params_network,<<"guest">>,<<"guest">>,
                                    <<"/">>,
                                    "slg1matt1.mgmt.xyz.com",5672,
                                    0,0,0,infinity,none,
                                    [#Fun<amqp_auth_mechanisms.plain.3>,
                                     #Fun<amqp_auth_mechanisms.amqplain.3>],
                                    [],[]},
                                <<"skytap">>,1000,1,1,none,none,none,
                                "upstream-slg"},
                            {resource,<<"/">>,exchange,<<"skytap">>}}]}},
                  {restart_type,{permanent,1}},
                  {shutdown,4294967295},
                  {child_type,worker}]

On Wed, Sep 12, 2012 at 4:51 PM, Matt Pietrek <mpietrek at skytap.com> wrote:

> I'm experience a problem where "rabbitmqctl stop" takes a very long time
> (5+ minutes, or never) shuts down.
>
> This is RabbitMQ 2.8.6 and Erlang R15B01 on Ubuntu 10.04.
>
> The broker is running with the Federation plug-in in use, FWIW.
>
> The last little bits of console output are:
>
> --------
> Mnesia(rabbit at mq1): write performed by {tid,307,<0.151.0>} on record:
>     {rabbit_durable_route,
>             {binding,
>                 {resource,<<"/">>,exchange,<<"skytap">>},
>                 <<"cmcmd.us-west.default">>,
>                 {resource,<<"/">>,queue,<<"cmcmd">>},
>                 []},
>             const}
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> {needs_dump,40}
> Mnesia(rabbit at mq1): delete_object performed by {dirty,<0.237.0>} on
> record:
>     {rabbit_listener,rabbit at mq1,amqp,"mq1.blah.com",
>                          {0,0,0,0},
>                          5672}
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> already_dumped
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> already_dumped
> Mnesia(rabbit at mq1): Transaction log dump initiated by time_threshold:
> already_dumped
> --------
>
> The rabbit at mq1.log file:
>
> -------
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
> Stopping Rabbit
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
>     application: rabbitmq_management
>     exited: stopped
>     type: permanent
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
>     application: rabbitmq_federation
>     exited: stopped
>     type: permanent
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
>     application: rabbitmq_management_agent
>     exited: stopped
>     type: permanent
>
> =INFO REPORT==== 12-Sep-2012::16:38:14 ===
> stopped TCP Listener on 0.0.0.0:5672
>
> =WARNING REPORT==== 12-Sep-2012::16:38:14 ===
> Connection (<0.299.0>) closing: internal error in channel (<0.306.0>):
> shutdown
> -------
>
> And the rabbitm at mq1-sasl log:
>
> --------
> =SUPERVISOR REPORT==== 12-Sep-2012::16:38:14 ===
>      Supervisor: {<0.304.0>,amqp_channel_sup}
>      Context:    child_terminated
>      Reason:     shutdown
>      Offender:   [{pid,<0.306.0>},
>                   {name,channel},
>                   {mfa,{amqp_channel,start_link,
>                                      [direct,<0.299.0>,1,<0.305.0>,
>                                       #Fun<amqp_channel_sup.0.24953263>]}},
>                   {restart_type,intrinsic},
>                   {shutdown,4294967295},
>                   {child_type,worker}]
>
>
> =SUPERVISOR REPORT==== 12-Sep-2012::16:38:14 ===
>      Supervisor: {<0.304.0>,amqp_channel_sup}
>      Context:    shutdown
>      Reason:     reached_max_restart_intensity
>      Offender:   [{pid,<0.306.0>},
>                   {name,channel},
>                   {mfa,{amqp_channel,start_link,
>                                      [direct,<0.299.0>,1,<0.305.0>,
>                                       #Fun<amqp_channel_sup.0.24953263>]}},
>                   {restart_type,intrinsic},
>                   {shutdown,4294967295},
>                   {child_type,worker}]
> --------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120912/99994bd7/attachment.htm>


More information about the rabbitmq-discuss mailing list