[rabbitmq-discuss] The rabbitmq-server stop command hangs

[Tim Watson]

Hi Liz,

When the rabbit is stuck shutting down, is the source host for shovelB and shovelB_realtime (server.local) accessible - i.e., you can ping it, telnet to the amqp port, etc? It seems these two shovels have never got properly started, and they're hung (still trying to establish connections/channels) when you're trying to shut down. We are filing a bug and looking at how to fix this, but it would be helpful for me to understand the topology so I can simulate this bug when producing a fix.

Cheers,
Tim  

On 29 Nov 2012, at 14:55, Elizabeth Liao wrote:

> Hi,
> 
> I've attached the trace out with the latest version.  The reason for our shovel set up is that the local server does not know the names of our machines.  So, we leave it to the machine itself to set up a shovel connection to the local server.
> 
> Liz
> ________________________________________
> From: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discuss-bounces at lists.rabbitmq.com] on behalf of Tim Watson [tim at rabbitmq.com]
> Sent: Thursday, November 29, 2012 3:47 AM
> To: Discussions about RabbitMQ
> Subject: Re: [rabbitmq-discuss] The rabbitmq-server stop command hangs
> 
> Hi Liz,
> 
> Would you mind running the trace (against a stuck rabbit) again please? You'll need to download the latest version of the tool from https://github.com/hyperthunk/suptree_inspect/downloads and run it like so:
> 
> $ escript suptree_inspect.beam rabbit@<host> rabbit_shovel_sup | tee output-shovel.log
> 
> Many thanks!
> 
> Tim
> 
> On 28 Nov 2012, at 22:44, Tim Watson wrote:
> 
>> Hi Liz,
>> 
>> On 28 Nov 2012, at 19:45, Elizabeth Liao wrote:
>> 
>>> I've attached the rabbitmq.config, rabbit.log, rabbit-sasl.log of the node that we're running.
>>> 
>>>> Are you clustered (I'm assuming yes) and have you got some mirrored queues set up (also assuming yes)?
>>> We don't have clustering set up so no mirrored queues either.
>>> 
>> 
>> Yes of course, I should've spotted that from the trace.
>> 
>>>> And can you get the shovel status ("rabbitmqctl eval
>>> 'rabbit_shovel_status:status().'") at the time the shutdown is hanging?
>>> 
>>> # rabbitmqctl eval 'rabbit_shovel_status:status().'
>>> [{shovelB_realtime,starting,{{2012,11,28},{14,9,20}}},
>>> {shovelB,starting,{{2012,11,28},{14,9,20}}},
>>> {shovelA,
>>>   {running,
>>>       {source,
>>>           {amqp_params_direct,<<"guest">>,none,<<"/">>,rabbit at TEST2,
>>>               none,[]}},
>>>       {destination,
>>>           {amqp_params_network,<<"user1">>,<<"pass1">>,<<"/">>,
>>>               "server.local",undefined,0,0,5,infinity,none,
>>>               [#Fun<amqp_uri.7.123484526>,#Fun<amqp_uri.7.123484526>],
>>>               [],[]}}},
>>>   {{2012,11,28},{14,9,25}}}]
>>> ...done.
>>> 
>> 
>> Well unsurprisingly, Matthias was right about Shovel. Looks like the shovel plugin is stuck - can you explain what your shovel configuration is meant to be doing? It seems shovelB_realtime and shovelB are both hung during startup - are these guys meant to be sending data or receiving it? And shovelA - is that using a direct (Erlang) connection to obtain data from rabbit at Test2 (the source) and send it to the local server (on which it is currently running) using a network based connection? I'm a bit confused about that setup - why not the other way around? - though I've not really spent much time looking at the shovel plugin tbh so it may be I'm missing something very obvious.
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> Email Confidentiality Notice
> 
> The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
> <output-shovel.log>_______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss