[rabbitmq-discuss] How to deal with duplicated msgs when using HA mirror?

Thu May 17 15:08:56 BST 2012

What I've been doing is keeping a log in memcached of what tasks were in
process, and then persisting those to some kind of non-volatile storage,
along with their completion status. If I get a message for the same task, I
check to see if another process was already handling it before starting it
on one of the consumers.

- Alex

On Thu, May 17, 2012 at 5:06 AM, Matthew Sackman <matthew at rabbitmq.com>wrote:

> Hi,
>
> On Thu, May 17, 2012 at 09:27:29AM +0800, 黄大鹏 wrote:
> > As the article <<High Availability in RabbitMQ: solving part of the
> > puzzle<
> http://www.rabbitmq.com/blog/2011/10/25/high-availability-in-rabbitmq-solving-part-of-the-puzzle/
> >>>
> > mentioned, when master breakdown,  the last ack maybe not sync to all
> > slave,so when a slave promote to be a new master, the msgs which actualy
> be
> > consumed maybe redelivered to the consumers.
> > Is there any way to  identify these duplicated messages? our  scenario
> is :
> > multi producers, multi queue (every consumer has a private queue ), a
> > direct exchange and a fanout exchange.
>
> Actually, msg duplication can happen via a number of means not involving
> HA at all. HA just adds a few more places where such duplication can
> occur.
>
> There is no automatic way: a msg that's has the "isDelivered" flag set
> on it *may* have already been delivered to another client, which is to
> say that if that flag isn't set, then you shouldn't have to test for
> duplication. But really, you're going to have to have some sort of
> client side cache of hashes of recently seen messages, or some such.
>
> I'm afraid I'm not aware of any off-the-shelf solution for this, but
> this problem crops up regularly so there should be a fair amount of
> material out there on this stuff.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120517/382d5d2f/attachment.htm>