[rabbitmq-discuss] Problem in high availability setup together with corosync

Fri Jan 27 17:49:05 GMT 2012

Hey there,

today we experimented with some high availability setup featuring 
RabbitMQ + HA-Queues and Corosync.  Currently we have this setup:

  * 2 nodes named mq01ha and mq02ha

Both nodes have ha-ip addresses that move to the other node that is not 
down so that all services can proceed, no matter to what node they are 
connected.

Our test scenario:

One producer is pushing to mq01ha and one consumer is consuming from 
mq01ha, the consumer is a bit slower so that messages pile up unacked.

Now we kill mq01ha by pusing the virtual power button (virtual maschines 
are great!), corosync moves the ip from mq01ha to mq02ha and the 
consumers and producers proceed with their work.  This works great and 
everything is fine again.

Now (and this is the part where I need your knowledge) we currently have 
the setup that if mq01 starts up again the ip address moves back to mq01.

Now mq01 is back up, all consumers and producers are proceeding with 
their work.  But unfortunately in mq01 and mq02 all messages that were 
unacked while mq01 gone down are still unacked and there is some not 
existing connection open in the connections view on rabbitmq. This is 
true for both, the web interface and for rabbitmqctl. Even if I shutdown 
all consumers and producers this connection exists, so it's not related 
to any real TCP connection.

I think that rabbitmq is not designed for this kind of scenario, but 
maybe you can tell me more about what rabbitmq does.  Maybe we found 
some bug, I don't know.

What we are now planning is a simple setup with only one ha-ip that 
moves from host to host only in failover so that - in theory - 
everything should work properly again.  But we have to check if then 
again we have the problem that unacked messages stay unacked.

Hopefully I described this stuff clear enough :)

Thanks!

Cheers,
Christopher.

-- 
Christopher Grebs
Data Processing Developer

Tel +49-30-611035-1664
Fax -49-30-611035-2652
christopher.grebs at native-instruments.de

NATIVE INSTRUMENTS GmbH
Schlesische Strasse 29-30
10997 Berlin, Germany

http://www.native-instruments.com

***

KOMPLETE 8 ULTIMATE - the premium NI producer collection
=>   http://www.native-instruments.com/komplete8

TRAKTOR KONTROL S2 - the professional 2.1 DJ system
=>   http://www.native-instruments.com/s2

->>>>>>  NATIVE INSTRUMENTS - The Future of Sound<<<<<<-

Registergericht: Amtsgericht Charlottenburg
Registernummer: HRB 72458
UST.-ID.-Nr. DE 20 374 7747

Geschäftsführung: Daniel Haver (CEO), Mate Galic