[rabbitmq-discuss] RabbitMQ hanging during 'rabbitmqctl stop' - please help

[Casey Marshall]

I'm using RabbitMQ federated exchanges among a number of servers. Each
server has upstream exchanges on each of the others. I don't want to
call it a cluster, since that has another meaning in RabbitMQ altogether
... let's call it a "federated group".

This worked fine over AMQP, both in my own local KVM-virtualized testing
environment, and it seemed to as well in Amazon EC2 with unencrypted
AMQP. However, I need to secure the connections in EC2 with AMQPS before
we go into production.

I was able to automate creation and distribution of certificates among
the servers, and the setup works fine in a local KVM-virtualized
environment. However, when I started testing it in EC2 across regions, I
started having problems. RabbitMQ hangs on some of the servers in EC2
during 'service rabbitmq-server restart'.

I have scripts to automate updating the federated exchanges, to effect
topology changes on the "federated group" -- adding or removing a
server, or changing its role (different roles have different exchange
upstreams). After this script updates the federation config, the
subsequent restart hangs on some of the servers.

What should I do to make my RabbitMQ "federated group" more robust?

I've attached:

1. rabbitmq.log and rabbitmq-sasl.log excerpt from one of the hosts that
persistently gets stuck during a RabbitMQ 'stop', near the time of the hang.
2. 'rabbitmqctl status' from that host.

Installed on the system:

ii  esl-erlang                       1:16.b.1-1~ubuntu~precise       Erlang
ii  rabbitmq-server                  3.1.3-1
AMQP server written in Erlang

Much thanks,
Casey
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit at ip-10-234-134-253.log
Type: text/x-log
Size: 23477 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130723/e5bd98be/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit at ip-10-234-134-253-sasl.log
Type: text/x-log
Size: 7398 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130723/e5bd98be/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: report.log
Type: text/x-log
Size: 16517 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130723/e5bd98be/attachment-0002.bin>