[rabbitmq-discuss] How to deal with duplicated msgs when using HA mirror?
alexander.ward at quibids.com
Thu May 17 15:08:56 BST 2012
What I've been doing is keeping a log in memcached of what tasks were in
process, and then persisting those to some kind of non-volatile storage,
along with their completion status. If I get a message for the same task, I
check to see if another process was already handling it before starting it
on one of the consumers.
On Thu, May 17, 2012 at 5:06 AM, Matthew Sackman <matthew at rabbitmq.com>wrote:
> On Thu, May 17, 2012 at 09:27:29AM +0800, 黄大鹏 wrote:
> > As the article <<High Availability in RabbitMQ: solving part of the
> > puzzle<
> > mentioned, when master breakdown, the last ack maybe not sync to all
> > slave,so when a slave promote to be a new master, the msgs which actualy
> > consumed maybe redelivered to the consumers.
> > Is there any way to identify these duplicated messages? our scenario
> is :
> > multi producers, multi queue (every consumer has a private queue ), a
> > direct exchange and a fanout exchange.
> Actually, msg duplication can happen via a number of means not involving
> HA at all. HA just adds a few more places where such duplication can
> There is no automatic way: a msg that's has the "isDelivered" flag set
> on it *may* have already been delivered to another client, which is to
> say that if that flag isn't set, then you shouldn't have to test for
> duplication. But really, you're going to have to have some sort of
> client side cache of hashes of recently seen messages, or some such.
> I'm afraid I'm not aware of any off-the-shelf solution for this, but
> this problem crops up regularly so there should be a fair amount of
> material out there on this stuff.
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss