[rabbitmq-discuss] Unique Messages in Queue

Fri Jul 31 19:41:34 BST 2009

Matthias,

Minimizing the probability of sending out a duplicate message is the
practical objective. So you are right in saying that the best we can
do is to make it very unlikely that a duplicate mail is sent out.

A filtering proxy would make most sense because I wanted to move away
from expensive I/O for persistence in the first place, plus if needed,
redundancy can be introduced later for fault-tolerance.

Thanks for all the help!

Best,

Vidit

On Fri, Jul 31, 2009 at 2:08 PM, Matthias Radestock<matthias at lshift.net> wrote:
> Vidit,
>
> Vidit Drolia wrote:
>>
>> The primary problem is that since the action being triggered by the
>> message is an email, I can't revert the action. So I am trying to
>> ensure that the application sending emails gets a message only once.
>> Is there another approach I can take to this problem?
>
> If you replace "process(msg)" in my last email with "send_email(msg)", you
> will see that what you are asking for is impossible. The best one can do (in
> any system, involving rabbit or not) is to make it *very unlikely* that an
> email is sent more than once. As long as we can agree on that, let's proceed
> ...
>
> If your main concern is removing the duplicates the senders can produce,
> then I suggest inserting a filtering proxy, i.e. a process that consumes
> messages from one queue, de-dups them and publishes the non-dups to another
> exchange.
>
> This process does need to keep some state, so, as you say, if it crashes and
> the state is lost then you may get some dups. The process is very simple
> though, so the likelihood of it crashing should be low. Given that we have
> established that there can be no 100% no-dup guarantee, is it really worth
> worrying about that? If the answer is yes, then persisting that state, or
> replicating it between several redundant nodes are possible options.
>
>
> Regards,
>
> Matthias.
>