[rabbitmq-discuss] Message durability and partition tolerence

[Simon MacMullen]

Hi Jonathan. Thanks for an interesting mail!

On 08/11/11 16:26, Jonathan Oliver wrote:
> Since the introduction of better clustering support and message queue
> failover in v2.6, my company is starting to consider moving to RabbitMQ.
> But we had a few questions about the roadmap.
>
> We are looking to ensure that a message is only considered received by
> an exchange from a producing app when at least two physical instances of
> Rabbit acknowledge the receipt of the message--similar to Riak's method.
> Furthermore, we are looking to ensure that a message is considered
> consumed (ack'd) when when at least one physical instance receives the
> ack, which may result in a message being delivered more than once.
> Essentially, we'd like to propose a configuration where Rabbit can fall
> on the AP side of CAP instead of the CA side.

This starts to sound quite a lot like federation. Especially the 
AP-as-opposed-to-CA part, which could almost have been taken from 
http://www.rabbitmq.com/blog/2011/06/22/federation-plugin-preview-release/ 
:-)

(Incidentally this is reminding me that federation really needs better 
documentation right now! Sigh...)

Where your vision differs is the "at least two physical instances" part. 
I always assumed that people would use federations-of-clusters to get HA 
with partition tolerance. But that does require that you can specify 
that some groups of nodes *won't* get partitioned from each other. Easy 
if you're thinking in terms of data centres linked by WANs, less easy in 
the cloud. Hmm.

...snip discussion of ordering...

> For example, if an instance of
> Rabbit goes offline under the current HA configuration, when it rejoins
> the cluster the queues are wiped clean. Yuck. So much for guaranteed
> delivery.

With our current HA queues any offline slave queue has to be wiped and 
restarted because it doesn't know what happened while it was down - but 
that's just the same as having to sync up a RAID after a similar event. 
In particular it doesn't mean that messages were lost - the whole point 
is that the queue has other mirrors which will keep it up.

Non-HA queues don't get wiped at rejoin, since we know nothing can have 
happened to them while they were down.

> While under some kind of partition-tolerant configuration, the
> queues wouldn't need to be cleaned. Instead, the consuming application
> would be responsible for message idempotency.
>
> At the same time, I realize that you guys might be bumping against
> restrictions in the AMQP standard, for example multiple consumers
> receiving copies of the message where one consumer ack's the message,
> while another nack's the message.  But that's something where the
> messaging system can't determine the proper course of action without
> knowing the business context. In that case, I would imaging it should
> present that nack back to the producer as it currently does.

One thing that the federation plugin does not attempt to do at all is 
make the resultant network look like a single broker as defined by AMQP 
(or in any other sense). As far as I can see this is the only thing to 
do for an AP system. Trying to look like "a broker" is going to imply 
global state, which... ugh. So one corollary is that things like 
mandatory can't work across federation.

> The bottom line in my question and proposal really has to do with
> message loss. If the idea is to *not lose messages*, I can see a
> partition-tolerant configuration bringing a lot to the table and where
> the consuming application must be resilient to duplicate messages and
> ordering, thus freeing up RabbitMQ from having to worry so much about that.

I wonder about message loss in an AP Rabbit, with your "at least two 
instances" constraint. (Using that term rather than federated to imply 
that we're not inherently talking about the current plugin here).

You're a producer. You connect to a node and send it a message. That's 
one instance. So it needs to forward the message on to another instance 
before it can send you a confirm...

...but we need to cope with partitions. What happens if you and the node 
you're talking to are on a network partition? How long should it wait? 
If the answer is "forever" then I think you just lost Availability. If 
it isn't, then the partition can always heal just after you timed out, 
which I guess also means you weren't Available. How should this work?

FTR the current federation will confirm your message as soon as the 
first node has it, and will forward it when connectivity is available.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware