[rabbitmq-discuss] Implementing a "processing" queue with message expiry

[Marek Majkowski]

On Tue, Jan 11, 2011 at 00:15, Sami Samhuri <sami at samhuri.net> wrote:
> The system is pretty simple and will only have 2 persistent queues, 1
> producer, and 1 consumer (initially anyway). Our frontend machine will act
> on requests from our users and queue up jobs on request. The worker machine
> will get jobs from that queue, process them, and then place the results in a
> 2nd queue which the frontend machine will consume, making the results
> available to our users. So far it's pretty straightforward and RabbitMQ
> handles this with ease.
> ,----------,                    ,--------,
> |          |  --- Queue #1 ---> |        |   Work to be done goes in Queue #1
> | Frontend |                    | Worker |
> |          | <--- Queue #2 ---  |        |   Completed work goes in Queue #2
> `----------`                    `--------`
> Now real life has to come along and mess it all up. When a worker is struck
> by lightning in the simple system above the job it was working on would be
> lost, which is bad. To me it seems like we need a 3rd "queue" that holds
> jobs that are currently being processed, and a timeout so that if a job
> being processed hasn't completed within N minutes it's put back into the
> first queue so another worker can pick up the job.

That's a strange idea. Rabbitmq deals with this problem by using 'acks'.
A worker first gets a message (request), does the job, replies,
and after sending the reply it 'acks' the initial request message.
The request is removed from Rabbit only after it received an ack.

That way when your worker dies during processing of the request,
the request will be redelivered and hopefully handled by another worker.

> This queue must also be
> persistent. The problem is that this 3rd thing isn't really a queue at all.
> I'd like to remove any specific item from this thing, whether the job was
> completed or timed out, and thus I need to find and remove arbitrary
> messages/items.

Rabbit can't do it - you can't 'find arbitrary messages'. Rabbit queues
are FIFO-like structures. You append new messages to one end.
You pop messages from the other end.

> I feel like I need to write a bit of code that handles the list of
> jobs-in-progress and persists it to our main data store (distributed, backed
> up, etc). If I do all of that when not just put the other 2 queues in our
> main data store as well? Rabbit still buys us a little for those 2 queues,
> but not very much. It's easy enough for me to add authenticated HTTP
> endpoints and distributed, fault-tolerance persistence for a couple of
> queues.
> Is there some other solution that makes this easier, or does Rabbit have
> something to handle these sorts of cases? If I can't use Rabbit for all of
> this it may not be worth adding Rabbit to our infrastructure for the 2 very
> simple queues illustrated above.
> Thanks for reading this far :)

Alternative solution would be to make requests idempotent and re-send
the request when the client decides it's waiting too long (ie: message
was lost).

This is worse solution, in practice you can never assume how long
will it take to handle a request.

Cheers,
  Marek