[rabbitmq-discuss] Replicated message queue? DRDB+Hearbeat?

Fri Feb 22 03:26:12 GMT 2008

----- "Dmitriy Samovskiy" <dmitriy.samovskiy at cohesiveft.com> wrote:

> ... A bigger 
> question is how long it will take between a message is sent and the
> message is consumed. 
> Out of the box, I guess it depends on how fast you can restore the
> original broker (app, 
> system and/or network connectivity, depending on what failed).

  Yes, only you will be able to answer that.  In my case, the time to fix a unresponsive server is about an hour.

> >   I thought about using DRDB to replicate RabbitMQ's on-disk Mnesia
> store to a standby node.  I've used DRDB successfully for MySQL, and
> it works well.  And then use Hearbeat to start RabbitMQ on the standby
> node, which would then recover the Mnesia tables, and start running.
> So you are planning to produce to a broker and consume from the same
> broker. And once that 
> broker fails, you will have both producers and consumers shift to
> another broker, which 
> you will start on DRBD-cloned disk after the first broker failed?

  Yes.  DRDB keeps the master disk synced to the slave disk.  Hearbeat determines if the master fails, and switches the direction of the syncing, and can also move the virtual IP from the master to the slave.  Everything just connects to the virtual IP.  Obviously clients (producers and consumers) need to be robust, and have the ability to recovery from timeouts, and reconnect.

...
> application logic to identify duplicate messages (just in case).
...

  I've never come across a good way of doing this.  I can attach a unique id to each message, but then the clients have to keep state on which ids they've processed.  And if you multiple clients, they need to talk to each other too.  I don't want consumers to have to keep state.  

  Now depending on the message, of course, processing it twice may not matter that much.  For example, if the message is "Turn switch 2 On", then if you do that twice, it is harmless.  But obviously you don't want to processing every message twice just for high-availability.  

...
> Please let us know if you get to deploy rabbitmq with DRBD-backed
> storage and how it works 
> for you.

  I have to do a lot of testing of this configuration first.

> Thanks,
> Dmitriy

Tom