[rabbitmq-discuss] Unique Messages in Queue

[Darien Kindlund]

Hi Vidit,

To my knowledge, RabbitMQ doesn't provide this type of capability by
default.  However, you could easily write a "pre-filter" application,
which takes in duplicate messages from your source on one queue and
then outputs the non-duplicates out on another queue.  You don't need
a database or other expensive I/O to check for duplicates.

Presumably, you have a way of identifying unique messages; I'll assume
it's some sort of unique ID associated with the message...
You can have your "pre-filter" application use a bloom filter
(http://en.wikipedia.org/wiki/Bloom_filters), which allocates constant
size memory and a configurable false-positive percentage.
The only major issue with this approach, is that bloom filters hold a
maximal amount of entries (user configurable), so you'll have to deal
with the corner case of how the application should behave when it has
seen the maximum number of unique messages -- either recreate a new
bloom filter (and discard your previous history) or chain multiple
bloom filters together (slowly increasing your CPU costs).  Again, the
maximal value can be set exceedingly large (e.g., 10 million), in
order to reduce the (probable) likelihood of dealing with this
situation.

-- Darien

On Fri, Jul 31, 2009 at 11:39 AM, Vidit Drolia<viditdrolia at gmail.com> wrote:
> Hi,
>
> Is it possible to store unique messages in a queue? I have a message
> source that may provide duplicate messages but I do not want the
> message to be processed more than once. I am trying to avoid using a
> database or any other form of persistence in the client using the
> queues. Any help/advice will be very helpful.
>
> Thanks!
>
> Regads,
>
> --
> Vidit Drolia
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>