[rabbitmq-discuss] Message durability and partition tolerence
simon at rabbitmq.com
Mon Nov 14 15:14:04 GMT 2011
Hi Jonathan. Thanks for an interesting mail!
On 08/11/11 16:26, Jonathan Oliver wrote:
> Since the introduction of better clustering support and message queue
> failover in v2.6, my company is starting to consider moving to RabbitMQ.
> But we had a few questions about the roadmap.
> We are looking to ensure that a message is only considered received by
> an exchange from a producing app when at least two physical instances of
> Rabbit acknowledge the receipt of the message--similar to Riak's method.
> Furthermore, we are looking to ensure that a message is considered
> consumed (ack'd) when when at least one physical instance receives the
> ack, which may result in a message being delivered more than once.
> Essentially, we'd like to propose a configuration where Rabbit can fall
> on the AP side of CAP instead of the CA side.
This starts to sound quite a lot like federation. Especially the
AP-as-opposed-to-CA part, which could almost have been taken from
(Incidentally this is reminding me that federation really needs better
documentation right now! Sigh...)
Where your vision differs is the "at least two physical instances" part.
I always assumed that people would use federations-of-clusters to get HA
with partition tolerance. But that does require that you can specify
that some groups of nodes *won't* get partitioned from each other. Easy
if you're thinking in terms of data centres linked by WANs, less easy in
the cloud. Hmm.
...snip discussion of ordering...
> For example, if an instance of
> Rabbit goes offline under the current HA configuration, when it rejoins
> the cluster the queues are wiped clean. Yuck. So much for guaranteed
With our current HA queues any offline slave queue has to be wiped and
restarted because it doesn't know what happened while it was down - but
that's just the same as having to sync up a RAID after a similar event.
In particular it doesn't mean that messages were lost - the whole point
is that the queue has other mirrors which will keep it up.
Non-HA queues don't get wiped at rejoin, since we know nothing can have
happened to them while they were down.
> While under some kind of partition-tolerant configuration, the
> queues wouldn't need to be cleaned. Instead, the consuming application
> would be responsible for message idempotency.
> At the same time, I realize that you guys might be bumping against
> restrictions in the AMQP standard, for example multiple consumers
> receiving copies of the message where one consumer ack's the message,
> while another nack's the message. But that's something where the
> messaging system can't determine the proper course of action without
> knowing the business context. In that case, I would imaging it should
> present that nack back to the producer as it currently does.
One thing that the federation plugin does not attempt to do at all is
make the resultant network look like a single broker as defined by AMQP
(or in any other sense). As far as I can see this is the only thing to
do for an AP system. Trying to look like "a broker" is going to imply
global state, which... ugh. So one corollary is that things like
mandatory can't work across federation.
> The bottom line in my question and proposal really has to do with
> message loss. If the idea is to *not lose messages*, I can see a
> partition-tolerant configuration bringing a lot to the table and where
> the consuming application must be resilient to duplicate messages and
> ordering, thus freeing up RabbitMQ from having to worry so much about that.
I wonder about message loss in an AP Rabbit, with your "at least two
instances" constraint. (Using that term rather than federated to imply
that we're not inherently talking about the current plugin here).
You're a producer. You connect to a node and send it a message. That's
one instance. So it needs to forward the message on to another instance
before it can send you a confirm...
...but we need to cope with partitions. What happens if you and the node
you're talking to are on a network partition? How long should it wait?
If the answer is "forever" then I think you just lost Availability. If
it isn't, then the partition can always heal just after you timed out,
which I guess also means you weren't Available. How should this work?
FTR the current federation will confirm your message as soon as the
first node has it, and will forward it when connectivity is available.
More information about the rabbitmq-discuss