[rabbitmq-discuss] Unique Messages in Queue

Fri Jul 31 18:24:36 BST 2009

Thanks Darien and Matthias.

Each message received is actually triggering an email. I can use Bloom
Filters to make sure messages are not duplicate, but if the node
running the filter application fails, I will end up losing historical
information. Running the filter on multiple nodes may be a solution.

The primary problem is that since the action being triggered by the
message is an email, I can't revert the action. So I am trying to
ensure that the application sending emails gets a message only once.
Is there another approach I can take to this problem?

Thanks again!

Vidit

On Fri, Jul 31, 2009 at 1:03 PM, Matthias Radestock<matthias at lshift.net> wrote:
> Vidit,
>
> Vidit Drolia wrote:
>>
>> Is it possible to store unique messages in a queue? I have a message
>> source that may provide duplicate messages but I do not want the
>> message to be processed more than once.
>
> As Darien pointed out, deduplicating messages at the client end isn't all
> that hard, and even easier than he described if, say, you can guarantee that
> message ids are monotonically increasing.
>
> The really, really, hard part is ensuring that a message only gets
> *processed* once.
>
> When can a message be considered to have been processed? Let's assume we
> have an app that pulls messages off a rabbit queue and calls a function
> process(msg) to process them. At what point then has the message been
> processed? At the exact point we call the function? At the exact point it
> returns? Somewhere inbetween? Whatever point you choose, you then still have
> to *record* the fact that the point has been reached, so that the message
> can be forgotten for good, or, alternatively, if the point hasn't been
> reached, replayed at a later point. That act of recording - whether it be by
> acknowledging the message in rabbit, or some other means - itself can fail,
> which will result in eventual resending and thus duplication.
>
> The problem here is that the processing of the message itself, let alone the
> combination of it with the recording/acknowledging action are not atomic.
>
> The only way to solve this is to either make everything - rabbit, your app,
> any apps it talks to, etc - part of a gigantic transaction - thus ensuring
> atomicity - or - and this is by far the easier and better option - construct
> your apps in such a way that messaging is idempotent.
>
>
> Regards,
>
> Matthias.
>

-- 
Vidit Drolia