I've done some more testing and knocked up two sample apps:<br><br>Publisher (loads up 100,000 items)<br><a href="http://pastebin.com/E0ixXP8c">http://pastebin.com/E0ixXP8c</a><br><br>Consumer<br><a href="http://pastebin.com/yAMcHkbu">http://pastebin.com/yAMcHkbu</a><br>
<br>If I run the publisher, then once that has enqueued 100,000 items run the consumer. If after it has consumed 20,000 and still consuming, I do a graceful start of the RabbitMQ broker's OS (running on Windows Server 2008, so Start, Shutdown/Restart), when the broker comes back online I have 100,000 items in the queue (even though approx 20,000 are consumed and acknowledged).<br>
<br>I built a test host running Ubuntu which seems to handle the graceful OS restarts and will report the queue size (approx 80,000).<br><br>If I do a dirty shutdown (remove power, forcefully stop VM, etc) neither Windows or Ubuntu have persisted to disc but I assume that may be related to it not being committed to disk at that stage.<br>
<br>Kind Regards,<br><br>Ben<br><br><div class="gmail_quote">On Fri, Nov 11, 2011 at 5:44 PM, Ben Lewis <span dir="ltr"><<a href="mailto:stuckinwigan@gmail.com">stuckinwigan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">Hi <span>Emile,<br><br>Thanks for the response. Here's the output with from the rabbitmqctl command at different stages:<br><br>First, with me ack'ing the messages:<br><br>Prior to starting consumer<br>
------</span><span>------</span><span>------</span><span>------</span><br><span>Listing queues ...<br>Test.ItemProcess 67645 0<br>...done.<br><br>As consumer is consuming, just reboot rebooting broker<br>
</span><span>------</span><span>------</span><span>------</span><span>------</span><br><span>Listing queues ...<br>Test.ItemProcess 42936 1<br>...done.<br><br>After broker rebooted and RabbitMQ started<br>
</span><span>------</span><span>------</span><span>------</span><span>------</span><br><span>Listing queues ...<br>Test.ItemProcess 67645 0<br>...done.<br><br>With
noack=true the messages are delivered to the client quicker than I can
reboot the broker, but when it comes back up the queue is empty with 0
unacknowledged messages.<br>
<br>
</span>I'm not using transactions (I put a seperate console app together
to test this) and I do want to persist and acknowledge the message.
I've stepped through my code in debug and its hitting the BasicAck
method.<br>
<br>Kind Regards,<br><br>Ben<br><br></div><div class="gmail_quote"><div class="im">On Fri, Nov 11, 2011 at 4:10 PM, Emile Joubert <span dir="ltr"><<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>></span> wrote:<br>
</div><div><div class="adm"><div id="q_13393cdd995816e2_3" class="ajR h4"><div class="ajT"></div></div></div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Ben,<br>
<div><br>
On 11/11/11 15:12, Ben Lewis wrote:<br>
> I have got a direct exchange feeding a durable queue, if I load say<br>
> 50,000 messages onto the queue and then start a consumer to work through<br>
> those item's.<br>
><br>
> My consumer creates the connection, then creates the model (I'm using<br>
> .NET) and sets the Qos prefetch count to 1, then calls BasicGet,<br>
> processes the message, sends an ack (BasicAck), then enumerates until<br>
> BasicGet returns null. Finally disposing the model and closing the<br>
> connection.<br>
<br>
</div>You don't need to set QoS if you retrieve messages synchronously<br>
(BasicGet). QoS makes sense with BasicConsume. That is not the cause of<br>
your trouble though.<br>
<div><br>
> If I restart the host I have the RabbitMQ broker running on (Windows)<br>
> after processing and ack'ing 40,000 messages while my consumer is<br>
> working through the remaining items, when the RabbitMQ broker comes back<br>
> online the 40,000 messages are redelivered (even though they were<br>
> acknowledged).<br>
><br>
> Is this supposed to happen?<br>
<br>
</div>No. Acknowledged messages are forgotten about by the broker, so you<br>
should not be seeing them again. What is the output of<br>
<br>
rabbitmqctl list_queues name messages_ready messages_unacknowledged<br>
<br>
before and after the restart? Is it possible that you are somehow not<br>
acknowledging the messages? Do you get the same result if you use the<br>
noAck flag of BasicGet?<br>
<br>
If you are using transactions and failing to commit the transaction then<br>
you could see apparently acknowledged message reappearing - is that a<br>
possibility?<br>
<div><br>
> If this is by design, any suggestions or advice on how I can minimise<br>
> the number of redelivered messages if the broker dies/restarts?<br>
<br>
</div>If you don't care about persisting messages then you can publish the<br>
messages in non-persistent mode (set the delivery mode in the basic<br>
properties to 1).<br>
<span><font color="#888888"><br>
<br>
-Emile<br>
</font></span></blockquote></div></div></div><br>
</blockquote></div><br>