[rabbitmq-discuss] RabbitMQ rendered non-operational after killing 1 node of 3 h/a

[Eugene Kirpichov]

Hi,

I have a cluster of 3 rabbits (version 2.7.1), and a bunch of services
that connect to it and declare a bunch of H/A queues. Some of the
queues are exclusive, some aren't.

Some of the services are connected to rabbit01 and some to rabbit03.
Nobody's connected to rabbit02.

Then I kill rabbit02, and, after a short while (but not immediately),
rabbitmqctl list_queues, rabbitmqctl list_connections and rabbitmqctl
list_channels completely stop working - they just hang.

After these start hanging, I killed all applications that were
connected to RabbitMQ, and then I killed rabbit03 so only rabbit01
remained. But those rabbitmqctl commands were still hanging. However,
if I restart rabbit01 and bring up the other cluster nodes, everything
is working again.

It seems like there was some internal error that lead to death of a
mirrored queue coordinator process, or something... I'm attaching the
log from rabbit01.

-- 
Eugene Kirpichov
http://www.linkedin.com/in/eugenekirpichov
We're hiring! http://tinyurl.com/mirantis-openstack-engineer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit at control01.log
Type: application/octet-stream
Size: 24844 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121017/cb02d087/attachment.obj>