[rabbitmq-discuss] The rabbitmq-server stop command hangs

Tim Watson tim at rabbitmq.com
Tue Nov 13 11:34:39 GMT 2012


Hi Elizabeth

The timeout indicates that the shutdown sequence is stuck, which is expected, although it is clearly not stuck shutting down the channels which is good to know. The reason you can't connect using suptree_inspect is less clear. Are you using the same account to launch both executables? The cookie (usually stored in $HOME/.erlang.cookie file) needs to be the same. When the test2 node is online, can you execute *this* command successfully

$ erl_call -n test2 at test2 -a 'erlang get_cookie []'

? If so then suptree_inspect *should* work - we can try this out in the console like so, making sure that suptree_inspect.beam is in the current directory. First we'll make sure that connectivity works (you'll see an error if it doesn't)

$ erl -sname debug -s init stop -eval 'true = net_kernel:connect(rabbit at iske).' -noshell

then we can run suptree_inspect manually (this *shouldn't* be required)

$ erl
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false] [lock-counting]

Eshell V5.9.1  (abort with ^G)
1> code:ensure_loaded(suptree_inspect).
{module,suptree_inspect}
2> suptree_inspect:main(["test2 at test2", "rabbit_sup"]).

Whichever way you do this, the stuck node needs to be running and contactable over rpc (the cookies need to be the same).

Let me know how you get on please.

Cheers,
Tim

On 9 Nov 2012, at 19:07, Elizabeth Liao wrote:

> Tim,
> 
> Here's what I got:
> 
> # rabbitmqctl status
> Status of node test2 at test2 ...
> 
> # rabbitmqctl eval 'process_info(whereis(rabbit_channel_sup_sup)).'
> Error: {badarg,[{erlang,process_info,[undefined],[]},
>                {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,572}]},
>                {rpc,'-handle_call_call/6-fun-0-',5,
>                     [{file,"rpc.erl"},{line,203}]}]}
> 
> # rabbitmqctl eval 'application:which_applications().'
> Error: {timeout,{gen_server,call,[application_controller,which_applications]}}
> 
> #  escript suptree_inspect.beam test2 at test2 rabbit_sup | tee output.log
> ERROR: unable to contact test2 at test2
> 
> #  escript suptree_inspect.beam rabbit at test2 rabbit_sup | tee output.log
> ERROR: unable to contact rabbit at test2
> 
> Liz
> 
> ________________________________________
> From: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discuss-bounces at lists.rabbitmq.com] on behalf of Tim Watson [tim at rabbitmq.com]
> Sent: Thursday, November 08, 2012 8:04 PM
> To: Discussions about RabbitMQ
> Subject: Re: [rabbitmq-discuss] The rabbitmq-server stop command hangs
> 
> Liz,
> 
> Ok - what does `rabbitmqctl eval 'application:which_applications().' look like?
> 
> Also, can you grab https://github.com/downloads/hyperthunk/suptree_inspect/suptree_inspect.beam and run it like so:
> 
>  $ escript suptree_inspect.beam rabbit at host rabbit_sup | tee output.log
> 
> Obviously replacing rabbit at host with the correct node name. This should put a bunch of noise into output.log which I'd like to see! Please feel free to scrub any sensitive data before posting, though I very much doubt there will be any.
> 
> That tool btw was hacked together very quickly and compiled against R15B01, so it *may* bomb out on you, in which case we'll fall back to doing some more 'eval' until we can figure out why your rabbit is stuck.
> 
> Cheers,
> Tim
> 
> On 8 Nov 2012, at 18:42, Elizabeth Liao wrote:
> 
>> Hi Tim,
>> 
>> I finally reproduced an instance where rabbitmq-server appears to hang.  Here's the the output of the command:
>> #  rabbitmqctl eval 'process_info(whereis(rabbit_channel_sup_sup)).'
>> Error: {badarg,[{erlang,process_info,[undefined],[]},
>>               {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,572}]},
>>               {rpc,'-handle_call_call/6-fun-0-',5,
>>                    [{file,"rpc.erl"},{line,203}]}]}
>> 
>> Liz
>> 
>> ________________________________________
>> From: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discuss-bounces at lists.rabbitmq.com] on behalf of Tim Watson [tim at rabbitmq.com]
>> Sent: Thursday, November 08, 2012 3:34 AM
>> To: Discussions about RabbitMQ
>> Subject: Re: [rabbitmq-discuss] The rabbitmq-server stop command hangs
>> 
>> Liz,
>> 
>> On 7 Nov 2012, at 18:55, Tim Watson wrote:
>> 
>>> Hi Liz,
>>> 
>>> On 11/07/2012 02:49 PM, Elizabeth Liao wrote:
>>>> Hi,
>>>> 
>>>> I still haven't been able to get this working.  Anyone have any suggests to diagnose the problem?
>>> 
>>> Sorry we haven't got back to you sooner. I think some investigation on a stuck node will be necessary. If you can give me ssh access to the machine where the shutdown is hanging then I can do this for you, otherwise I'll write up a module you can compile and execute using escript and send it to you. I have actually started on that latter idea anyway, but it's getting a bit late now so I'll probably finish it off in the morning! :)
>>> 
>> 
>> Actually come to think of it, we could check a couple of things briefly using rabbitmqctl first. Can you send the output of this please: rabbitmqctl eval 'process_info(whereis(rabbit_channel_sup_sup)).'
>> 
>>> Cheers,
>>> Tim
>>> 
>>>> Thanks a lot!
>>>> 
>>>> Liz
>>>> ________________________________________
>>>> From: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discuss-bounces at lists.rabbitmq.com] on behalf of Elizabeth Liao [eliao at seegrid.com]
>>>> Sent: Monday, November 05, 2012 3:48 PM
>>>> To: rabbitmq-discuss at lists.rabbitmq.com
>>>> Subject: [rabbitmq-discuss] rabbitmq-server stop hangs
>>>> 
>>>> Hi,
>>>> 
>>>> I am currently running rabbitmq-server-2.8.7 and erlang-R15B on FC11.  I'm having a problem where rabbit sometimes hangs on this command:
>>>> /etc/init.d/rabbitmq-server stop such that rabbitmqctl status hangs as well with this output:
>>>> Status of node rabbit at test1 ...
>>>> 
>>>> The only error report that I see in in the rabbit at test1.log file is:
>>>> =ERROR REPORT==== 5-Nov-2012::16:23:51 ===
>>>> connection<0.526.0>, channel 5 - error:
>>>> {amqp_error,channel_error,"expected 'channel.open'",'basic.ack'}
>>>> 
>>>> I don't see anything logged in the rabbit at test1-sasl.log file.
>>>> 
>>>> I realize that this isn't very much information to go on so if anyone could offer more insight into this problem, it would be greatly appreciated!
>>>> 
>>>> Thanks.
>>>> 
>>>> Liz
>>>> 
>>>> 
>>>> Email Confidentiality Notice
>>>> 
>>>> The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
>>>> _______________________________________________
>>>> rabbitmq-discuss mailing list
>>>> rabbitmq-discuss at lists.rabbitmq.com
>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>> Email Confidentiality Notice
>>>> 
>>>> The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
>>>> _______________________________________________
>>>> rabbitmq-discuss mailing list
>>>> rabbitmq-discuss at lists.rabbitmq.com
>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> 
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> 
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> Email Confidentiality Notice
>> 
>> The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> Email Confidentiality Notice
> 
> The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



More information about the rabbitmq-discuss mailing list