[rabbitmq-discuss] The rabbitmq-server stop command hangs

[Tim Watson]

Hi Liz,

Can you please confirm a couple of things for us: what plugins are enabled in your setup? Are you clustered (I'm assuming yes) and have you got some mirrored queues set up (also assuming yes)?

Would you be able to post the rabbit.log and rabbit-sasl.log for all your nodes please? 

I'm not 100% sure what is causing this to hang, but I can see a few things in the trace data that make me suspicious.

The rabbit_reader seems to be attempting to consume data, and appears to be in 'start_connection' at shutdown time. I notice that both reader and writer processes are running using the rabbit_framing_amqp_0_8 layer - do you have AMQP-0.8 clients connected to the broker?

The 'rabbit_reader:start_connection' trace looks a bit suspect too:

stacktrace: [{rabbit_net,recv,2,[]},
             {rabbit_reader,mainloop,2,[]},
             {rabbit_reader,start_connection,7,[]},
             {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]

I wonder if a handshake has got stuck somehow? The rabbit_net:recv/2 implementation does result in a blocking receive. 

The only other things I wonder about are that there appears to be one queue process with a ton of data in its state record, but that doesn't necessarily mean anything if the supervision hierarchy shutdown hasn't reached that point yet.

It might take some time to get to the bottom of this issue just by looking at traces, but having the logs would help and obviously if you are able to get us access to the machine where your rabbit is stuck, that would help too. If necessary, vmware might be willing to sign some kind of NDA with regards access to your machine/data, if that helps at all.

Cheers,
Tim

On 27 Nov 2012, at 00:17, Elizabeth Liao wrote:

> Here's the output:
> 
> # rabbitmqctl eval 'app_utils:app_dependency_order([os_mon, mnesia,rabbit] ++ rabbit_plugins:active(),true).'
> [mnesia,mochiweb,amqp_client,webmachine,rabbitmq_mochiweb,os_mon,rabbit,
> rabbitmq_management_agent,rabbitmq_shovel,rabbitmq_management]
> 
> As for access to the machine, it's doubtful but I'll ask.
> 
> Liz
> ________________________________________
> From: Matthias Radestock [matthias at rabbitmq.com]
> Sent: Monday, November 26, 2012 6:47 PM
> To: Discussions about RabbitMQ
> Cc: Elizabeth Liao
> Subject: Re: [rabbitmq-discuss] The rabbitmq-server stop command hangs
> 
> Liz,
> 
> On 26/11/12 22:19, Elizabeth Liao wrote:
>> I've attached the inspect trace when using 3.0.
> 
> Thanks.
> 
> When your rabbit is running normally, i.e. it's not stuck in a shutdown,
> what does
> 
> rabbitmqctl eval 'app_utils:app_dependency_order([os_mon, mnesia,
> rabbit] ++ rabbit_plugins:active(),true).'
> 
> (all on one line)
> 
> return?
> 
> 
> Also, is there any way you could give Tim or me access to the machine
> while you have a stuck rabbit?
> 
> 
> Regards,
> 
> Matthias.
> Email Confidentiality Notice
> 
> The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss