[rabbitmq-discuss] Unique Messages in Queue

Fri Jul 31 18:03:29 BST 2009

Vidit,

Vidit Drolia wrote:
> Is it possible to store unique messages in a queue? I have a message
> source that may provide duplicate messages but I do not want the
> message to be processed more than once.

As Darien pointed out, deduplicating messages at the client end isn't 
all that hard, and even easier than he described if, say, you can 
guarantee that message ids are monotonically increasing.

The really, really, hard part is ensuring that a message only gets 
*processed* once.

When can a message be considered to have been processed? Let's assume we 
have an app that pulls messages off a rabbit queue and calls a function 
process(msg) to process them. At what point then has the message been 
processed? At the exact point we call the function? At the exact point 
it returns? Somewhere inbetween? Whatever point you choose, you then 
still have to *record* the fact that the point has been reached, so that 
the message can be forgotten for good, or, alternatively, if the point 
hasn't been reached, replayed at a later point. That act of recording - 
whether it be by acknowledging the message in rabbit, or some other 
means - itself can fail, which will result in eventual resending and 
thus duplication.

The problem here is that the processing of the message itself, let alone 
the combination of it with the recording/acknowledging action are not 
atomic.

The only way to solve this is to either make everything - rabbit, your 
app, any apps it talks to, etc - part of a gigantic transaction - thus 
ensuring atomicity - or - and this is by far the easier and better 
option - construct your apps in such a way that messaging is idempotent.

Regards,

Matthias.