[rabbitmq-discuss] Atomic acking of a specific set of messages

[peter at vagaband.co]

I'm trying to use RabbitMQ with Storm (for processing a stream of messages) 
using Trident (a high level abstraction on top of Storm). In order to gain 
exactly-once processing of messages with Storm, I need to be able to create 
a Storm spout that can retrieve a batch of messages from RabbitMQ and 
ack/fail them as a batch as an atomic unit. While amqp protocol or rabbitmq 
specific extensions to it does not seem to support this in an obvious 
manner, is there some pattern I could implement using what RabbitMQ 
provides to achieve this?

The flag to indicate you're acking multiple messages will not work as that 
limits you to work with just one batch, right? And you cannot pick and 
choose which batch of messages you can or cannot ack with that mode (i.e. 
batch 2 might succeed, batch 3 might fail and batch 4 might succeed. so if 
you ack batch 4, everything will be acked).

Can I use a consistent hashing exchange (or something to the effect by 
writing my own exchange plugin) to a select a batch of messages and route 
them to a specific queue and once all the messages in that queue (i.e. the 
queue is the batch), then the queue is discarded?

The key here is treat a batch of messages as an atomic unit so if a single 
message consumer node dies or all message consumer nodes die or a rabbitmq 
node dies, then the whole batch can be failed to be re-tried later.

Thoughts? 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130707/9d8d390b/attachment.htm>