[rabbitmq-discuss] Atomic acking of a specific set of messages

[Tim Watson]

We're about to start working on a message groups feature that might make doing this possible, but there's nothing in place like that at the moment.

Just out of interest, how would you anticipate ACKs working in light of this feature? For example, would we consider the whole group ack'ed if the last message had been ack'ed? If you're requiring the whole group to be delivered before the ack, you'd need the prefetch count (i.e., control over the max allowed un-ack'ed messages for a single consumer) to match up, which would mean the client has to know how many messages the group will contain up front (doesn't sound workable to me) or the client would have to keep bumping the prefetch count each time it is exceeded - also unworkable IMHO.

Also, you'd need to be able to recognise the last message in a group. A better design IMO would be to ack all messages as they arrive (and are processed) and by default only redeliver un-ack'ed messages for the group if a client disconnects while the group is still open. A configuration parameter (or runtime policy) on the exchange you're publishing to might indicate that instead of the default behaviour, we want to republish the whole group (including previously ack'ed messages) if the consumer goes away prematurely, but that will incur a space and time hit on the server as we'll need to publish the messages to an additional internal store (i.e., temporary queue) in order to be able to redeliver them later on - since ack'ed messages will normally disappear once they're ack'ed.

As you can see, there are various trade offs to make in designing this kind of feature. Feel free to share any insights into your own use case!

Cheers,
Tim

On 7 Jul 2013, at 19:10, "peter at vagaband.co" <peter at vagaband.co> wrote:

> I'm trying to use RabbitMQ with Storm (for processing a stream of messages) using Trident (a high level abstraction on top of Storm). In order to gain exactly-once processing of messages with Storm, I need to be able to create a Storm spout that can retrieve a batch of messages from RabbitMQ and ack/fail them as a batch as an atomic unit. While amqp protocol or rabbitmq specific extensions to it does not seem to support this in an obvious manner, is there some pattern I could implement using what RabbitMQ provides to achieve this?
> 
> The flag to indicate you're acking multiple messages will not work as that limits you to work with just one batch, right? And you cannot pick and choose which batch of messages you can or cannot ack with that mode (i.e. batch 2 might succeed, batch 3 might fail and batch 4 might succeed. so if you ack batch 4, everything will be acked).
> 
> Can I use a consistent hashing exchange (or something to the effect by writing my own exchange plugin) to a select a batch of messages and route them to a specific queue and once all the messages in that queue (i.e. the queue is the batch), then the queue is discarded?
> 
> The key here is treat a batch of messages as an atomic unit so if a single message consumer node dies or all message consumer nodes die or a rabbitmq node dies, then the whole batch can be failed to be re-tried later.
> 
> Thoughts? 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss