[rabbitmq-discuss] Exchange Feature request: Drop Duplicates

[Matthias Reik]

Even though it sounds like a nice feature, it is probably difficult to 
really implement, if not done on the client side. The duplicates might 
happen when delivering to the client side. but on the client side it 
should be quite easy to do the filtering:
* get a message from the queue,
* check against memcached (couchbase, or some other cache technology) 
whether the messageID exists.
* Add the new message to memcached (can be done with the previous step)
* Set the timeout in memcached to your window size.

This should be straight forward, would scale up to quite a lot of 
messages) and should remove (depending on your window size) all duplicates.

Is there a good reason why you wouldn't want to do this on the client 
side as described?

Cheers
Matthias

PS: as a caching technology you could of course do your own 
in-memory-solution but that's probably more work than to use an 
out-of-the-box solution.

On 2013-11-11 12:35 , Laing, Michael wrote:
> In our scenarios, messages are ultimately delivered to a 'retail' 
> rabbitmq instance to be delivered to a client. The pipelines that 
> process and deliver messages are purposefully redundant, hence there 
> may be multiple replicas of each message 'racing' to the endpoint.
>
> Usually, the replicas are resolved before getting to the retail 
> rabbit. When components fail, however, duplicates can leak through 
> during a small window of time. We eliminate those duplicates at the 
> retail layer by looking at each message_id. Ultimately, our client 
> contract allows duplicates as well in case one slips by.
>
> It seems to me that this is a generic issue.
>
> What would be useful in our case, and hopefully for many others, would 
> be a 'Duplicate Message ID Window' in milliseconds, as an exchange 
> attribute.
>
> If non-zero, the exchange would drop any message with a duplicate 
> message_id that appeared within the specified window of time, possibly 
> routing it to the alternate exchange, if set.
>
> In our case, a window of a few seconds, perhaps up to a minute would 
> suffice.
>
> Thanks,
>
> Michael
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131111/7fd1f05c/attachment.htm>