[rabbitmq-discuss] Failed to configure Rabbitmq HA(pacemaker)

[Jae Sang Lee]

Hi, all.


I read a rabbitmq HA(pacemaker) article at a homepage, then I tried to
configure two-node HA.
I succeeded run drbd and rabbitmq service by pacemaker. But It failed when
I tried to migrate from master to slave server.


This is first situation.
my node 'ha-node-1', 'ha-node-2'.
Master is 'ha-node-2'

ha-node-2# crm_mon
============
Last updated: Thu Dec 29 10:13:15 2011
Stack: openais
Current DC: ha-node-1 - partition with quorum
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
2 Nodes configured, 2 expected votes
3 Resources configured.
============

Online: [ ha-node-1 ha-node-2 ]

 Master/Slave Set: drbd_ms
     Masters: [ ha-node-2 ]
     Slaves: [ ha-node-1 ]
drbd_fs (ocf::heartbeat:Filesystem):    Started ha-node-2
bunny   (ocf::rabbitmq:rabbitmq-server):        Started ha-node-2


ha-node-2~# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root at glance-demo,
2011-12-26 13:05:30

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
    ns:486 nr:5 dw:491 dr:287 al:6 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0

I tested fail situation from master. so
ha-node-2~# crm node standby

ha-node-1~# crm_mon
============
Last updated: Thu Dec 29 10:32:56 2011
Stack: openais
Current DC: ha-node-1 - partition with quorum
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
2 Nodes configured, 2 expected votes
3 Resources configured.
============

Node ha-node-2: standby
Online: [ ha-node-1 ]

 Master/Slave Set: drbd_ms
     Masters: [ ha-node-1 ]
     Stopped: [ drbd:1 ]
drbd_fs (ocf::heartbeat:Filesystem):    Started ha-node-1

Failed actions:
    bunny_start_0 (node=ha-node-1, call=14, rc=1, status=complete): unknown
error

drbd is migrate to slave successfully, but resource bunny(rabbitmq) isn't
start.
this is rabbitmq error log

root at ha-node-1:/var/log/rabbitmq# cat startup_log
Activating RabbitMQ plugins ...
ERROR: Could not delete dir
/media/drbd1/rabbit at localhost-plugins-expand({cannot_delete,

"/media/drbd1/rabbit at localhost-plugins-expand/rabbit.script",

eacces})

I found slave's rabbitmq mnesia dir's owner is ais, not rabbitmq.
root at ha-node-1:/var/log/rabbitmq# cd /media/drbd1/
root at ha-node-1:/media/drbd1# ll
total 20
drwxr-xr-x 5 ais  ais   1024 2011-12-28 17:09 ./
drwxr-xr-x 3 root root  4096 2011-12-27 13:22 ../
-rw-r--r-- 1 ais  ais      3 2011-12-28 16:48 hello
drwx------ 2 ais  ais  12288 2011-12-28 16:47 lost+found/
drwxr-xr-x 5 ais  ais   1024 2011-12-29 10:32 rabbit at localhost/
drwxr-xr-x 2 ais  ais   1024 2011-12-28 17:09
rabbit at localhost-plugins-expand/

When ha-node-2 success to run rabbitmq,
mnesia dir's owner is rabbitmq.
root at ha-node-2:/media/drbd1# ll
total 20
drwxr-xr-x 5 rabbitmq rabbitmq  1024 2011-12-28 17:09 ./
drwxr-xr-x 3 root     root      4096 2011-12-27 10:58 ../
-rw-r--r-- 1 rabbitmq rabbitmq     3 2011-12-28 16:48 hello
drwx------ 2 rabbitmq rabbitmq 12288 2011-12-28 16:47 lost+found/
drwxr-xr-x 5 rabbitmq rabbitmq  1024 2011-12-29 09:42 rabbit at localhost/
drwxr-xr-x 2 rabbitmq rabbitmq  1024 2011-12-28 17:09
rabbit at localhost-plugins-expand/


Why pacemaker change owner of file/folder from rabbitmq to ais? I think
that is a reason to fail to run rabbitmq at slave(ha-node-1) server.

How can I fix that? I hope your reply.

Thanks.



ps.
this is pacemaker configure.
# crm configure show
node ha-node-1 \
        attributes standby="off"
node ha-node-2 \
        attributes standby="on"
primitive bunny ocf:rabbitmq:rabbitmq-server \
        params mnesia_base="/media/drbd1" \
        meta target-role="Started" is-managed="true"
primitive drbd ocf:linbit:drbd \
        params drbd_resource="drbd1" \
        op monitor interval="60s"
primitive drbd_fs ocf:heartbeat:Filesystem \
        params device="/dev/drbd1" directory="/media/drbd1" fstype="ext3"
ms drbd_ms drbd \
        meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true"
colocation bunny_on_fs inf: bunny drbd_fs
colocation fs_on_drbd inf: drbd_fs drbd_ms:Master
order bunny_after_fs inf: drbd_fs bunny
order fs_after_drbd inf: drbd_ms:promote drbd_fs:start
property $id="cib-bootstrap-options" \
        dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="2" \
        stonith-enabled="false" \
        no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
        resource-stickiness="100"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111229/755bde9e/attachment.htm>