[rabbitmq-discuss] Unique Messages in Queue
Matthias Radestock
matthias at lshift.net
Fri Jul 31 18:03:29 BST 2009
Vidit,
Vidit Drolia wrote:
> Is it possible to store unique messages in a queue? I have a message
> source that may provide duplicate messages but I do not want the
> message to be processed more than once.
As Darien pointed out, deduplicating messages at the client end isn't
all that hard, and even easier than he described if, say, you can
guarantee that message ids are monotonically increasing.
The really, really, hard part is ensuring that a message only gets
*processed* once.
When can a message be considered to have been processed? Let's assume we
have an app that pulls messages off a rabbit queue and calls a function
process(msg) to process them. At what point then has the message been
processed? At the exact point we call the function? At the exact point
it returns? Somewhere inbetween? Whatever point you choose, you then
still have to *record* the fact that the point has been reached, so that
the message can be forgotten for good, or, alternatively, if the point
hasn't been reached, replayed at a later point. That act of recording -
whether it be by acknowledging the message in rabbit, or some other
means - itself can fail, which will result in eventual resending and
thus duplication.
The problem here is that the processing of the message itself, let alone
the combination of it with the recording/acknowledging action are not
atomic.
The only way to solve this is to either make everything - rabbit, your
app, any apps it talks to, etc - part of a gigantic transaction - thus
ensuring atomicity - or - and this is by far the easier and better
option - construct your apps in such a way that messaging is idempotent.
Regards,
Matthias.
More information about the rabbitmq-discuss
mailing list