[rabbitmq-discuss] Shovel restart command hanging

[Elizabeth Liao]

Hi,

Currently we are running Rabbitmq 2.8.2 and are running into some problems with a shovel configuration.  We were running into the shovel wedge issues that was mentioned in another discussion on this mailing list. It appeared to be sometimes resolved by the command: rabbitmqctl eval 'application:stop(rabbitmq_shovel), application:start(rabbitmq_shovel).'  However, other sometimes, executing this command would hang.  

Our other observations were as follows:
  * Executing all rabbit commands were hang (rabbitmqctl status, /etc/init.d/rabbitmq-server status). A control-c breaks out of it
  * Rabbitmq continues to handle all messages  
  * Error messages from logs below
  * I have all all logs and the contents of the MNESIA_BASE directory if needed
 
We noticed that 2.8.6 RabbitMQ contains shovel fixes. Would the new version of RabbitMQ prevent the problems that we're seeing - we know that it will prevent a shovel from getting stuck in the terminated state but is something else also going on?  and if not, what is the recommended fix? 

Thanks!

Liz
---------------
Errors from <nodename>.log:
=ERROR REPORT==== 27-Aug-2012::15:53:17 ===
** Last message in was socket_closed
** When Server state == {state,amqp_network_connection,
                            {state,#Port<0.211307>,5,<0.11430.8>,131072,
                                undefined,false},
                            <0.11417.8>,<0.11420.8>,
                            {amqp_params_network,<<"user1">>,
                                <<"pass1">>,<<"/">>,
                                "host1",5672,0,0,5,infinity,
                                none,
                                [#Fun<amqp_auth_mechanisms.plain.3>,
                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
                                [],[]},
                            0,
                            [{<<"capabilities">>,table,
                              [{<<"publisher_confirms">>,bool,true},
                               {<<"exchange_exchange_bindings">>,bool,true},
                               {<<"basic.nack">>,bool,true},
                               {<<"consumer_cancel_notify">>,bool,true}]},
                             {<<"copyright">>,longstr,
                              <<"Copyright (C) 2007-2012 VMware, Inc.">>},
                             {<<"information">>,longstr,
                              <<"Licensed under the MPL.  See http://www.rabbitmq.com/">>},
                             {<<"platform">>,longstr,<<"Erlang/OTP">>},
                             {<<"product">>,longstr,<<"RabbitMQ">>},
                             {<<"version">>,longstr,<<"2.8.2">>}],
                            #Fun<amqp_connection_sup.0.39273983>,
                            #Fun<amqp_connection_sup.2.54430129>,false}
** Reason for termination == 
** socket_closed_unexpectedly

Output from <nodname>-sasl.log:

=CRASH REPORT==== 27-Aug-2012::15:53:17 ===
  crasher:
    pid: <0.11418.8>
    registered_name: []
    exception exit: socket_closed_unexpectedly
      in function  gen_server:terminate/6
    initial call: amqp_gen_connection:init/1
    ancestors: [<0.11417.8>,amqp_sup,<0.201.0>]
    messages: []
    links: [<0.11416.8>,<0.11417.8>,#Port<0.211307>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 23
    reductions: 782
  neighbours:

=SUPERVISOR REPORT==== 27-Aug-2012::15:53:17 ===
     Supervisor: {<0.11417.8>,amqp_connection_sup}
     Context:    child_terminated
     Reason:     socket_closed_unexpectedly
     Offender:   [{pid,<0.11418.8>},
                  {name,connection},
                  {mfa,
                      {amqp_gen_connection,start_link,
                          [amqp_network_connection,
                           {amqp_params_network,<<"user1">>,
                               <<"pass1">>,<<"/">>,
                               "host1",5672,0,0,5,infinity,
                               none,
                               [#Fun<amqp_auth_mechanisms.plain.3>,
                                #Fun<amqp_auth_mechanisms.amqplain.3>],
                               [],[]},
                           #Fun<amqp_connection_sup.0.39273983>,
                           #Fun<amqp_connection_sup.2.54430129>,[]]}},
                  {restart_type,intrinsic},
                  {shutdown,brutal_kill},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 27-Aug-2012::15:53:17 ===
     Supervisor: {<0.11417.8>,amqp_connection_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.11418.8>},
                  {name,connection},
                  {mfa,
                      {amqp_gen_connection,start_link,
                          [amqp_network_connection,
                           {amqp_params_network,<<"user1">>,
                               <<"pass1">>,<<"/">>,
                               "host1",5672,0,0,5,infinity,
                               none,
                               [#Fun<amqp_auth_mechanisms.plain.3>,
                                #Fun<amqp_auth_mechanisms.amqplain.3>],
                               [],[]},
                           #Fun<amqp_connection_sup.0.39273983>,
                           #Fun<amqp_connection_sup.2.54430129>,[]]}},
                  {restart_type,intrinsic},
                  {shutdown,brutal_kill},
                  {child_type,worker}]