[rabbitmq-discuss] Web use scenarios

[Bill Moseley]

On Tue, Jan 4, 2011 at 2:12 PM, andrew chase <andrew at glyde.com> wrote:

> Most of your bullet points are outside the realm (IMO) of what a
> message system should be concerned with. In particular, it looks like
> most of the features you're after are more in line with what a
> background job processing system would handle. Those types of systems
> often use messaging under the covers but often also use some type of
> persistent data store in addition (or instead of) a messaging system.
> I'd suggest taking a look at a couple of these systems to see if they
> may be more what you're after (I know a couple ruby-based ones of the
> top of my head like Resque, DelayedJob, etc
> https://github.com/defunkt/resque).
>

Thanks Andrew,

I spent some time looking at those tonight and came full circle.  Found a
post where Resque was used as a replacement for DJ and then in the post's
comments people suggesting RabbitMQ... ;)  I'll keep reading, though.  I
have plenty to learn.

I think our organization has been in the habit of building somewhat
monolithic code and are looking for a solution that does everything as we do
it now.  I'm concerned we will attempt to build our own if we can't find a
perfect fit, which I suspect is a much harder task to get right than we
think.  As for scheduling, we already have a database table with the
scheduled items, so maybe we only need a few producers that run from cron.


I had two slightly more RabbitMQ-specific questions that are hopefully more
straight forward to answer.

When a job is re-queued due to failure to ACK is there anything that
indicates to the next consumer that the message has already been delivered
to a failed consumer? I need a way to recognize that a message is causing
consumers to die and stop the job.

And is it a common pattern for consumers to use RabbitMQ to message back
that their work is done?  I was wondering if that's a good use for x-expires
and/or x-message-ttl.

Thanks again,


On Tue, Jan 4, 2011 at 1:39 PM, Bill Moseley <moseley at hank.org> wrote:
> > Anyone have time to take a few of these on?
> > I know they are not specifically on-topic (and a bit open-ended) but we
> are
> > evaluating messaging solutions (and even discussing writing our own) so
> it
> > would be a huge help to get some feedback.  I'd be happy to provide more
> > specifics if needed.
> > Thanks,
> >
> > On Mon, Jan 3, 2011 at 3:13 PM, Bill Moseley <moseley at hank.org> wrote:
> >>
> >> Hi,
> >> I'm just starting out with RabbitMQ and trying to wrap my head around a
> >> few concepts.  I apologize if I ask something that should be obvious or
> I
> >> missed in the docs and faq.  I'm hoping someone can help with a few
> >> scenarios, and please point me to further reading as needed.
> >> I'm curious about use-cases that are not directly related to RabbitMQ,
> but
> >> how to best use RabbitMQ in an web application's architecture.  Thes are
> >> probably very common situations.
> >> 1) Notification of job status and completion:  If the web app produces a
> >> request to generate an expensive PDF is there a common approach to
> detecting
> >> when the consumer is completed and the pdf is ready for downloading?
>  Should
> >> I create a single-use return queue with x-expires and/or x-message-ttl
> on
> >> the queue?   In a web app the web user may never come back to pick up
> the
> >> PDF.
> >> 2) Job sequencing:  What's a common approach to multiple step processing
> >> that must be done serially?  I.e. queue job "A" and when it's complete
> queue
> >> job "B".  Doesn't really seem like consumer "A" should be responsible
> for
> >> submitting job "B".  Is it better to create a consumer "AB" that submits
> "A"
> >> and waits for a message that "A" is complete and then submits job "B"?
> >> 3) Scheduling: I assume there's no facility for scheduling jobs to be
> >> delivered to consumers at some future time.  Is there a better approach
> than
> >> using cron to check the database for jobs that should be queued based on
> >> time?
> >> 4) Cancel a request:  We have some situations when a subsequent request
> >> should cancel processing of an earlier request.  (Frankly, I think this
> is a
> >> design problem on our end.)
> >> 5) Prevent duplicate job requests: In a web environment when things
> break
> >> problems often compound by users reloading requests that caused the
> problem
> >> in the first place.  I'm not sure RabbitMQ would be involved at all in
> this
> >> scenario, but I'm asking just in case. ;)
> >> Finally, it's great that RabbitMQ will send a message again if is is not
> >> ACK'd and a consumer closes its connection.  But, is there any kind of
> max
> >> retry count available?  That is, what's a good approach to prevent an
> errant
> >> job from killing all consumers one-by-one?
> >> Thanks,
> >> --
> >> Bill Moseley
> >> moseley at hank.org
> >
> >
> >
> > --
> > Bill Moseley
> > moseley at hank.org
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> >
>



-- 
Bill Moseley
moseley at hank.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110104/10cc6a80/attachment-0001.htm>