[rabbitmq-discuss] Queue replication (I know, I know)

Fri Oct 15 00:20:02 BST 2010

At my company we're taking a serious look at using AMQP to unify our
communications between various apps and sites (about 3000 locations in a
hub and spoke topology). Volume will be relatively low, but some of our
apps have requirements on delivery guarantees which we've worked hard to
meet by designing our own protocols and error handling in the past. One
of our goals here is to actually reduce the amount of error handling we
have to do for each app, so the delivery guarantees of AMQP are
important to us.

Of all of the brokers we've looked at, RabbitMQ seems the most
functional and most mature, but the lack of replicated queues is
problematic for us because the hardware that will be running the broker
in our locations is COTS non-enterprise-grade stuff (just the nature of
our business). To pile on to the issue, we're running Windows (ugh) so a
nice, simple solution like DRBD is ruled out.

We don't need HA or real-time failover, we just need to make sure that
persistent messages get stored on the broker and backed up somewhere
else so that if the broker server goes belly up we can somehow restore
what was in the queue at the time that the server failed. Server
failures happen regularly enough at our remote sites that we'd like a
backup, and we're prepared to deal with a complete loss of the data if
we lose both the server and its backup at the same time, but having a
backup of the data would allow us to save a significant amount of effort
for the common case.

I've been digging through the site (and even the source) to see what
options we might have. The fact that we're running this on Windows (not
my choice) seems to put us in a very small minority, so really I'm
looking for any kind of advice on how we might be able to pull this off.
I've seen some mention of replaying transaction logs in the mailing list
archives but I can't seem to find any real details on what they or how
to use them.

I'm hoping that somehow this is possible to accomplish because the
alternatives mean a lot more work for us :(

Thanks,

Derek