[rabbitmq-discuss] Implementing a "processing" queue with message expiry

[Steven Taylor]

 Often, a bit like threading, things can get split off, but later reconverge
/ reconvene / rejoin the same execution stream (e.g. do the resutls end up
in a database / are they sent to the same end client)

"idempotent" as mentioned is a very useful solution (i.e. the job could get
done twice, but results/persistent state will look the same). You could
implement that as reject 2nd duplicated worker results, or copy it in again,
it doesn't matter. Assuming subsystems underneath the worker are transaction
based (in the database sense), you can ensure nothing gets inconsistent.

It took me a a few days to get comfortable with the "idempotent" idea, but
now I think I can work that idea into most situations.  The resulting code
while not materially that different appears to me to be more complete, and
thinking through an additional edge case, once you get used to it, in my
opinion adds value and doesn't get in the way of the development process.

-Steven
On 11 January 2011 15:21, Derek Chen-Becker <dbecker at cpicorp.com> wrote:

> On 01/10/2011 06:15 PM, Sami Samhuri wrote:
> > Hi all,
> >
> > I'm designing a small distributed system with some queueing involved and
> > have some doubts that Rabbit is the right solution for parts of it.
> > Hoping that some of you can shed some light and let me know if Rabbit
> > can do it all, which would be great, or if there's a better and/or
> > simpler solution I've missed. I realize that Rabbit is for message
> > passing and is not a work queue, so perhaps I'm trying to shove a square
> > peg into a round hole. If so please tell me that I'm barking up the
> > wrong tree.
>
> We use RabbitMQ for a render farm work queue and the way you deal with
> possible "lost" consumers is with ACKs. You need to turn off auto-ack
> when you set up the consumer, and then after you process the message you
> explicitly ACK it (BasicAck). If, for some reason, your client dies
> while in the middle of processing messages, any unacked messages it was
> holding on to get re-inserted into the queue.
>
> Derek
>  _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110112/222d1f37/attachment.htm>