[rabbitmq-discuss] Evaluating RabbitMQ - a few questions

Wed Apr 29 22:40:58 BST 2009

Hello all,

I'm investigating using RabbitMQ as a replacement for a few db-backed work
queues.  Our replication master sees considerable bit of write traffic into
these tables, and it would be nice to take that off of their hands.  My
first attempt will be to replace our dirty users queue -- this stores a
simple list of user ids, indicating for which users we need to do some
background processing & push out an updated user profile.  We mark many as
'dirty' during the course of the day, and run several workers to update the
profiles.  We usually have several hundred thousand users in the queue at a
time.

This seems like a good fit for RabbitMQ, but I have a few
questions/constraints:

   - Our current system allows us to do a unique constraint on userid -- if
   someone is already in the dirty queue and dirties his or her profile again,
   we don't want to update it multiple times. We do an 'insert ignore', leaving
   us with at most one row per user.  Are there any clever ways to emulate this
   behavior?

   I can think of a few ways to work around this outside of the queueing
   system, but all involve storing state elsewhere. I'd love a native solution
   that would allow me to fire as many "this user is dirty" messages into the
   queue, but guarantee my workers that the message message I read from a queue
   encapsulates all previous dirtying events.  Perhaps this is a pipe dream.

   - Our workers like to leak memory, so we often find ourselves killing
   them ungracefully.  While missing a few dirty profiles isn't terrible, it
   would be nice premature worker death didn't have to cause this.  Along these
   lines, I'm thinking we'd receive the message, do work, and then ack recieipt
   of the message.  If we die before the ack, the message lives on and someone
   else will clean that profile later.

   My question is about ack timing: how long does the worker have before the
   message is requeued? is this time configurable? is it configurable per
   queue?

I'm looking forward to hearing (er, reading) your thoughts.

-Brendan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090429/bccf3f87/attachment.htm 