[rabbitmq-discuss] Prevent queue flooding

Sun Oct 30 05:55:47 GMT 2011

I have two usage questions;

First, we have customers submitting jobs (via a web app) at a pretty even
rate, and as such any individual customer's job gets done on a pretty
regular basis.  But, it's possible for a single customer to flood the queue
with requests, which leads to everyone else waiting on that one customer's
jobs to complete.  I would like it to reduce the effect of one customer's
actions on everyone else.

My plan was to use Memcached to track submission rates per customer and
when they exceed some rate publish their jobs to a secondary (low priority)
queue.  Then assign fewer workers to the low priority queue.

Is this a good approach?

Second, when the web app queues a message (to process some "backend" job)
the web app sets a "pending" state in the database.  What normally happens
is one worker processes the job, then this worker sends a new message that
a second worker picks up and completes the job on the web app side.  In the
most simple case "completing" the job means the state in the database is
changed from "pending" to "complete".

Is this a common use pattern?

My question is how to handle failures and detect stale jobs.  For example,
what if the first worker dies and never sends the completed (or failed)
message to the second worker?  The database is then left indicating
"pending".   I would tend to use cron for this. That is, have cron look for
old "pending" states and either re-queue the request or mark it as failed
(say after a few failed retry attempts).

Are there other approaches I should consider?

Thanks,

-- 
Bill Moseley
moseley at hank.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111030/c1cd2eb3/attachment.htm>