[rabbitmq-discuss] Implementing a "processing" queue with message expiry

[Sami Samhuri]

Hi all,

I'm designing a small distributed system with some queueing involved and
have some doubts that Rabbit is the right solution for parts of it. Hoping
that some of you can shed some light and let me know if Rabbit can do it
all, which would be great, or if there's a better and/or simpler solution
I've missed. I realize that Rabbit is for message passing and is not a work
queue, so perhaps I'm trying to shove a square peg into a round hole. If so
please tell me that I'm barking up the wrong tree.

The system is pretty simple and will only have 2 persistent queues, 1
producer, and 1 consumer (initially anyway). Our frontend machine will act
on requests from our users and queue up jobs on request. The worker machine
will get jobs from that queue, process them, and then place the results in a
2nd queue which the frontend machine will consume, making the results
available to our users. So far it's pretty straightforward and RabbitMQ
handles this with ease.

,----------,                    ,--------,
|          |  --- Queue #1 ---> |        |   Work to be done goes in Queue
#1
| Frontend |                    | Worker |
|          | <--- Queue #2 ---  |        |   Completed work goes in Queue #2
`----------`                    `--------`

Now real life has to come along and mess it all up. When a worker is struck
by lightning in the simple system above the job it was working on would be
lost, which is bad. To me it seems like we need a 3rd "queue" that holds
jobs that are currently being processed, and a timeout so that if a job
being processed hasn't completed within N minutes it's put back into the
first queue so another worker can pick up the job. This queue must also be
persistent. The problem is that this 3rd thing isn't really a queue at all.
I'd like to remove any specific item from this thing, whether the job was
completed or timed out, and thus I need to find and remove arbitrary
messages/items.

I feel like I need to write a bit of code that handles the list of
jobs-in-progress and persists it to our main data store (distributed, backed
up, etc). If I do all of that when not just put the other 2 queues in our
main data store as well? Rabbit still buys us a little for those 2 queues,
but not very much. It's easy enough for me to add authenticated HTTP
endpoints and distributed, fault-tolerance persistence for a couple of
queues.

Is there some other solution that makes this easier, or does Rabbit have
something to handle these sorts of cases? If I can't use Rabbit for all of
this it may not be worth adding Rabbit to our infrastructure for the 2 very
simple queues illustrated above.

Thanks for reading this far :)

-s
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110110/2ce3b615/attachment.htm>