<div class="gmail_quote">On Fri, Nov 6, 2009 at 2:43 AM, Matthew Sackman <span dir="ltr"><<a href="mailto:matthew@lshift.net">matthew@lshift.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">On Fri, Nov 06, 2009 at 10:06:20AM +0000, Matthew Sackman wrote:<br>
> That's fine. I have to say that it's unlikely this patch will make it<br>
> through - the memory management code has gone through a lot of change<br>
> recently as we're getting a much better handle on resource management.<br></div></blockquote><div>Agreed. This is a definitely a workaround fix for the problem. In the interest of full disclosure, I have gotten rabbitmq to crash with this patch for the same reason, by getting the memory to spike before excess can be collected, so this isn't a full fix by any means. I will try to dig further into the root cause in 1.7.0 release when I have time.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">
> Whilst you've obviously been working from the head of our default branch<br>
> (many thanks!), there are a couple of issues with garbage collecting<br>
> every process like that, for example, it's possible that garbage<br>
> collecting vast numbers of processes will take longer than the<br>
> memory_check_interval, making messages queue up for the memory manager<br>
> process. This would become a problem if the garbage collection is unable<br>
> to reclaim any memory at all - eg millions of queues, all of which are<br>
> empty.<br></div></blockquote><div>However, with rabbit's current memory problems, I wouldn't run it with more than 10 queues, let alone millions ;). I will give the java client test a try.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">
<br>
</div>Some immediate ideas to improve this a little.<br>
1) Only do the GC when you initially hit the memory alarm. I.e. in the<br>
first case when going from non-alarmed to alarmed, put the gc in there,<br>
then maybe recurse again (though you'll likely want another param on the<br>
function to stop infinite recursion).<br></blockquote><div>This was my initial thought as well. However, by the time the alarm goes off, it is often too late for this to stop rabbit from crashing. For example, the default memory alarm is set to 0.95. Many of the crashes where to due to a failed allocation of 200-300 MB by the VM. At 4GB (rounding *up*), 200 MB is the limit of the allocation that can be made before garbage collecting (1 - .2/4 = 0.95). So, if more is allocated, the server is dead in the water.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
2) Only put GC in processes that are known to eat lots of RAM. Eg if<br>
it's the persister, then putting in a manual GC right after it does a<br>
snapshot is probably a good idea.<br></blockquote><div>The persister is indeed where the binary memory is hanging around, but I am not sure if the snapshot point is the problem. The crashes happen when adding batches into the queue. Its like the persister can't keep up. <br>
<br>For now, I will try this at the same point shown in the patch, but only do the persister:<br><br>garbage_collect(whereis(rabbit_persister)).<br><br>I would say the correct pattern would be something like python's MemoryError or handling of a malloc failure in C. Is there an exception that is thrown, where a garbage collect can be run? </div>
</div><br>_steve<br>