<div><div>I have two usage questions;</div><div><br></div><div>First, we have customers submitting jobs (via a web app) at a pretty even rate, and as such any individual customer's job gets done on a pretty regular basis. But, it's possible for a single customer to flood the queue with requests, which leads to everyone else waiting on that one customer's jobs to complete. I would like it to reduce the effect of one customer's actions on everyone else.</div>
<div><br></div><div>My plan was to use Memcached to track submission rates per customer and when they exceed some rate publish their jobs to a secondary (low priority) queue. Then assign fewer workers to the low priority queue.</div>
<div><br></div><div>Is this a good approach?</div><div><br></div><div><br></div><div>Second, when the web app queues a message (to process some "backend" job) the web app sets a "pending" state in the database. What normally happens is one worker processes the job, then this worker sends a new message that a second worker picks up and completes the job on the web app side. In the most simple case "completing" the job means the state in the database is changed from "pending" to "complete".</div>
</div><div><br></div><div>Is this a common use pattern?</div><div><br></div><div>My question is how to handle failures and detect stale jobs. For example, what if the first worker dies and never sends the completed (or failed) message to the second worker? The database is then left indicating "pending". I would tend to use cron for this. That is, have cron look for old "pending" states and either re-queue the request or mark it as failed (say after a few failed retry attempts).</div>
<div><br></div><div>Are there other approaches I should consider?</div><div><br></div><div>Thanks,</div><div><br></div><div><br></div><div><div><br></div>-- <br>Bill Moseley<br><a href="mailto:moseley@hank.org" target="_blank">moseley@hank.org</a><br>
</div>