[rabbitmq-discuss] Prevent queue flooding

[Emile Joubert]

Hi Bill,

On 30/10/11 05:55, Bill Moseley wrote:
> My plan was to use Memcached to track submission rates per customer and
> when they exceed some rate publish their jobs to a secondary (low
> priority) queue.  Then assign fewer workers to the low priority queue.

Another possible approach is to publish all jobs for each customer to
their own queue. This gives you the opportunity to allocate resources
more fairly, but it does entail a bit more complexity. You will still
need to publish job notifications in a master jobs queue, to enable
workers to find the customer-specific queues.

> Second, when the web app queues a message (to process some "backend"
> job) the web app sets a "pending" state in the database.  What normally
> happens is one worker processes the job, then this worker sends a new
> message that a second worker picks up and completes the job on the web
> app side.  In the most simple case "completing" the job means the state
> in the database is changed from "pending" to "complete".
> 
> Is this a common use pattern?

I'm not sure how common that is, because it sounds very specific to the
needs of your application.

> My question is how to handle failures and detect stale jobs.  For
> example, what if the first worker dies and never sends the completed (or
> failed) message to the second worker?  The database is then left
> indicating "pending".   I would tend to use cron for this. That is, have
> cron look for old "pending" states and either re-queue the request or
> mark it as failed (say after a few failed retry attempts).
> 
> Are there other approaches I should consider?

You could use acknowledgements. Workers should only acknowledge a job
once it is complete. If the worker fails before completion and drops the
connection to the broker then unacknowledged messages will be requeued
for other workers to pick up.



-Emile