<div><div>I have two usage questions;</div><div><br></div><div>First, we have customers submitting jobs (via a web app) at a�pretty even rate, and as such any individual customer's job gets done on a pretty regular basis. �But, it's possible for a single customer to flood the queue with requests, which leads to everyone else waiting on that one customer's jobs to complete. �I would like it to reduce the effect of one customer's actions on everyone else.</div>
<div><br></div><div>My plan was to use Memcached to track submission rates per customer and when they exceed some rate publish their jobs to a secondary (low priority) queue. �Then assign fewer workers to the low priority queue.</div>
<div><br></div><div>Is this a good approach?</div><div><br></div><div><br></div><div>Second, when the web app queues a message (to process some "backend" job) the web app sets a "pending" state in the database. �What normally happens is one worker processes the job, then this worker sends a new message that a second worker picks up and completes the job on the web app side. �In the most simple case "completing" the job means the state in the database is changed from "pending" to "complete".</div>
</div><div><br></div><div>Is this a common use pattern?</div><div><br></div><div>My question is how to handle failures and detect stale jobs. �For example, what if the first worker dies and never sends the completed (or failed) message to the second worker? �The database is then left indicating "pending". � I would tend to use cron for this. That is, have cron look for old "pending" states and either re-queue the request or mark it as failed (say after a few failed retry attempts).</div>
<div><br></div><div>Are there other approaches I should consider?</div><div><br></div><div>Thanks,</div><div><br></div><div><br></div><div><div><br></div>-- <br>Bill Moseley<br><a href="mailto:moseley@hank.org" target="_blank">moseley@hank.org</a><br>
</div>