Hi, John:<br><br><div class="gmail_quote">On Fri, Nov 2, 2012 at 7:19 PM, John Cartwright <span dir="ltr"><<a href="mailto:john.c.cartwright@noaa.gov" target="_blank">john.c.cartwright@noaa.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
The example suggests that the worker wait until the processing is<br>
complete to send the ACK for a given message. �So if my processing<br>
takes many hours, is this a problem for RabbitMQ to have the worker<br>
process consume a message but wait for a very long time to acknowledge<br>
it?<br></blockquote><div><br></div><div>The answer is that it depends. �:-)</div><div><br></div><div>While your message is in the "consumed but unACKed by the consumer" state, the broker is holding on to a copy of the message to guard against the possible failure of your consumer (which will be detected by seeing the connection close unexpectedly) by redelivering in the future if necessary.</div>
<div><br></div><div>If the message was published with persistence requested it may be sitting on disk during this period. �Even if it wasn't published persistently, but the broker came under memory pressure, the message may still have been swapped out to disk. �In both cases, keeping that message alive will eat some disk space. �It will also eat some memory, both for the message itself, assuming it hasn't been swapped out due to memory pressure, and a tiny bit of bookkeeping overhead.</div>
<div><br></div><div>Whether that becomes a problem or not while your long running consumer is chewing on the message depends on just what sort of resources are available in your broker (RAM and disk particularly) couner-balanced against just how many other messages are passing through the broker, and potentially also languishing in this state for a long time.</div>
<div><br></div><div>Rabbit can queue and persist messages pending their delivery and acknowledgment, but it's best not used deliberately as a long term store of data. �A handy mantra is that "A happy Rabbit is an empty Rabbit." �In other words, Rabbit (and other messaging systems) runs best when it's able to deliver messages in a reasonably timely way, although if things block up due to crashed or slow consumers or the like, mechanisms exist to keep things going, as long as you don't resource exhaust the universe.</div>
<div>�</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Is there a better way to handle this scenario?<br></blockquote><div><br></div><div>Again, it depends. �If you have a good sense for how much of your traffic is going to result in these long-working consumers, and you can bound that to some reasonable extent to avoid having vast piles of messages in the broker, delivered but waiting to be ACKed, then you might be just fine doing things the way you describe. �</div>
<div><br></div><div>If you expect that you'll have a ton of these slow-to-ACK messages, you may want to consider a slightly more elaborate messaging structure, say where the consumers ACK the message as soon as they've persisted a local copy to work on, and use another queue to notify any upstream services or publishers who care that the work in question has been finished.</div>
<div><br></div><div>Then again, perhaps the publisher side doesn't need to be directly informed that consumer work is done, in which case your workflow on the consumer side would be to consume the message, persist it to local storage so that you have a safe copy of it you can resume work upon if your consumer dies, and then ACK it, thereby accepting responsibility for the message's contents from the broker and allowing the broker to forget about the message and stop worrying about having to redeliver it.</div>
<div><br></div><div>Make sense?</div><div><br></div><div>Jerry</div><div><br></div></div>