[rabbitmq-discuss] Problem in high availability setup together with corosync

[Christopher Grebs]

Hey there,

I just missed to say that we are running RabbitMQ 2.7.1.

Regards,
Christopher.

On 01/27/2012 06:49 PM, Christopher Grebs wrote:
> Hey there,
>
> today we experimented with some high availability setup featuring 
> RabbitMQ + HA-Queues and Corosync.  Currently we have this setup:
>
>  * 2 nodes named mq01ha and mq02ha
>
> Both nodes have ha-ip addresses that move to the other node that is 
> not down so that all services can proceed, no matter to what node they 
> are connected.
>
> Our test scenario:
>
> One producer is pushing to mq01ha and one consumer is consuming from 
> mq01ha, the consumer is a bit slower so that messages pile up unacked.
>
> Now we kill mq01ha by pusing the virtual power button (virtual 
> maschines are great!), corosync moves the ip from mq01ha to mq02ha and 
> the consumers and producers proceed with their work.  This works great 
> and everything is fine again.
>
> Now (and this is the part where I need your knowledge) we currently 
> have the setup that if mq01 starts up again the ip address moves back 
> to mq01.
>
> Now mq01 is back up, all consumers and producers are proceeding with 
> their work.  But unfortunately in mq01 and mq02 all messages that were 
> unacked while mq01 gone down are still unacked and there is some not 
> existing connection open in the connections view on rabbitmq. This is 
> true for both, the web interface and for rabbitmqctl. Even if I 
> shutdown all consumers and producers this connection exists, so it's 
> not related to any real TCP connection.
>
> I think that rabbitmq is not designed for this kind of scenario, but 
> maybe you can tell me more about what rabbitmq does.  Maybe we found 
> some bug, I don't know.
>
> What we are now planning is a simple setup with only one ha-ip that 
> moves from host to host only in failover so that - in theory - 
> everything should work properly again.  But we have to check if then 
> again we have the problem that unacked messages stay unacked.
>
> Hopefully I described this stuff clear enough :)
>
>
> Thanks!
>
> Cheers,
> Christopher.
>


-- 
Christopher Grebs
Data Processing Developer

Tel +49-30-611035-1664
Fax -49-30-611035-2652
christopher.grebs at native-instruments.de

NATIVE INSTRUMENTS GmbH
Schlesische Strasse 29-30
10997 Berlin, Germany

http://www.native-instruments.com



***

KOMPLETE 8 ULTIMATE - the premium NI producer collection
=>   http://www.native-instruments.com/komplete8

TRAKTOR KONTROL S2 - the professional 2.1 DJ system
=>   http://www.native-instruments.com/s2

->>>>>>  NATIVE INSTRUMENTS - The Future of Sound<<<<<<-

Registergericht: Amtsgericht Charlottenburg
Registernummer: HRB 72458
UST.-ID.-Nr. DE 20 374 7747

Geschäftsführung: Daniel Haver (CEO), Mate Galic