[rabbitmq-discuss] Message ack timeout

[Dmitriy Samovskiy]

Hi Haldun,

Haldun Bayhantopcu wrote:
> Hi all,
> 
> I'm trying to set up a system where one process produces several
> messages to many consumer processes. Each consumer creates a new
> record in database for each message. Since this is a bit slow,
> and heavy work for database when especially a lot of messages
> are produced in a short interval, I considered buffering the
> messages in consumer processes, and then sending them to the

Interesting problem. I personally can think of 2 ways how you can design it to meet your 
requirements. (Please note it's purely on theoretical level, I haven't tested any of it).


1. Your consumer can establish a connection to rabbitmq broker, and then set up N 
channels. Do basic.get on each channel, you will end up with a batch of at most N 
messages, write the batch to the database, and then ack each message individually. Then 
repeat.

A problem of course is that something may happen before you finish ack'ing all messages 
(say after you ack'ed 2 of 5). It is my understanding that you won't be able to ack them 
all together, since basic.ack can only ack messages on its channel. Not sure if you can 
handle that in your app, but unack'ed 3 messages will be delivered to you again in this 
scenario.


1a. You can use nearly identical approach by having your consumers open multiple 
connections to broker (and a single channel inside each connection). Might get more 
complicated, as you will have to juggle multiple sockets.


2. "buffering" is somewhat like "queueing", so why not use rabbitmq to be your buffer? In 
this case, you set up a regular consumer (basic.consume). When a message arrives, you 
publish it back to rabbitmq but to a different routing_key (say "finished.{batch_number}" 
to topic-based exchange), and then ack the original one. Once you accumulate N messages, 
you write entire batch to the database, and then send something like "END" to 
"finished.{batch_number}". Have another process regularly check all finished.* messages, 
and for those that were not properly END'ed, put them back into original queue.

In this case, your window of inconsistency should be much smaller I think. With multiple 
consumer, make sure to use "finished.{consumer_id}_{batch_number}" instead.



- Dmitriy