[rabbitmq-discuss] Monitoring delivery of async messages

[McIntosh Jason]

We publish a message back after processing a batch to say how much data was processes for that batch.  Then the original publisher consumes those messages and republishes anything that was missed.  It's not quite a two phased commit but it works for us,
Jason

Sent from my iPhone

> On May 19, 2014, at 5:09 AM, Michael Klishin <mklishin at gopivotal.com> wrote:
> 
> On 19 May 2014 at 13:08:33, Billy Stack (bs.stack at gmail.com) wrote:
>>> Due to the importance of performance, we do not have any "acking"  
>> going on - so all messages are effectively "fire and forget"
> 
>> We have a "reporting application" that subscribes to the messages  
>> and creates reports.
>> The problem is that these reports are also critical - i.e. we cannot  
>> afford to have messages that go missing/get dropped etc
>> 
>> So the question I have is how do we go about monitoring the all message  
>> data is in the reporting application i.e. what do people usually  
>> do in a scenario like this?
> 
> Use publisher confirms and delivery confirmation. Both (primarily the former)
> will have performance effect but that's just how things are in distributed
> systems: throughput is at odds with confirmations, which is the only way
> to know that the peer got your message.
> 
>> We were thinking of writing a database script that looks at the  
>> database of the "web application" and "reporting application"  
>> and compares results - but this certainly does not feel right!  
> 
> If you know how to perform repairs when the two get out of sync, this is not unreasonable. 
> --  
> MK  
> 
> Software Engineer, Pivotal/RabbitMQ
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss