<div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I've setup a simple consumer which pushes ~10kb/s into a Rabbit queue,<br>
</blockquote>
<br></div>
Presumably you mean a *producer*. Is that producer publishing the messages as fast as it can, or is it rate-limited?<br>
</blockquote><div><br>Yes, my bad, should have said 'producer'.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">And how did you measure the rate?<div class="Ih2E3d">
</div></blockquote><div><br>Monitoring network IO with collectl on the RabbitMQ box.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">the memory appears to be fluctuating all over the map.<br>
<br></div>
Large fluctuations in memory are nothing unusual in most garbage-collected languages.<div class="Ih2E3d"></div></blockquote><div><br>Sure, though order of magnitude? 300 working set to 3GB? You're probably right.. in which case, damn!<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d">Also, I've added a liberal swap partition to the box, but I'm still seeing same behavior as before: rabbit died at around 2.2GB in the perister.log.<br>
<br></div>
If the producer isn't rate-limited then message backlogs can build up in various places throughout the system, before they ever get to the persister. Furthermore, unless you use transactions, messages are persisted asynchronously. So consequently you may have many more messages in memory than have been written to disk.<br>
<font color="#888888">
</font></blockquote><div><br>Interesting, I thought the semantics of 'persistent' flag guaranteed that each messages has been committed to a journal?<br></div></div>