[rabbitmq-discuss] How to deal with duplicated msgs when using HA mirror?

Thu May 17 11:06:24 BST 2012

Hi,

On Thu, May 17, 2012 at 09:27:29AM +0800, 黄大鹏 wrote:
> As the article <<High Availability in RabbitMQ: solving part of the
> puzzle<http://www.rabbitmq.com/blog/2011/10/25/high-availability-in-rabbitmq-solving-part-of-the-puzzle/>>>
> mentioned, when master breakdown,  the last ack maybe not sync to all
> slave,so when a slave promote to be a new master, the msgs which actualy be
> consumed maybe redelivered to the consumers.
> Is there any way to  identify these duplicated messages? our  scenario is :
> multi producers, multi queue (every consumer has a private queue ), a
> direct exchange and a fanout exchange.

Actually, msg duplication can happen via a number of means not involving
HA at all. HA just adds a few more places where such duplication can
occur.

There is no automatic way: a msg that's has the "isDelivered" flag set
on it *may* have already been delivered to another client, which is to
say that if that flag isn't set, then you shouldn't have to test for
duplication. But really, you're going to have to have some sort of
client side cache of hashes of recently seen messages, or some such.

I'm afraid I'm not aware of any off-the-shelf solution for this, but
this problem crops up regularly so there should be a fair amount of
material out there on this stuff.

Matthew