<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Thanks for the answers Matthew, couple of comments inline<div><br><div><div><div>On 2010-03-26, at 2:18 PM, Matthew Sackman wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Howdy,<br><br>On Fri, Mar 26, 2010 at 09:33:25AM -0500, Brendan Doyle wrote:<br><blockquote type="cite">1. If the gen_server was started by a supervisor, why would it not come back up and start accepting connections once again?<br></blockquote><br>I would expect it did. However, supervisors have a restart limit in<br>terms of the number of restarts within a particular period of time. I<br>would expect that it came back up and then immediately died again with<br>the same error, cycled like this for a while, and then the supervisor<br>gave up, exited, and took down the rest of rabbit. If this was the case<br>you should have entries in your logs about "Reached maximum restart<br>intensity".<br><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" color="#144FAE"><br></font></font></div></blockquote><blockquote type="cite"><div>Or, Erlang could have decided to try and reread the .beam file for the<br>module (no idea when it decides to do that), hit the same emfile error<br>and decided to die.<br><br></div></blockquote><div><br></div><div>I did look for the restart intensity message originally because that's what I would have expected to see, but the emfile error was the first and only error message other than connection open/close and persister logs rolling. So it was probably scenario #2 you describe. I just would have expected that to take down the whole VM if the child spec was defined with restart type permanent</div><br><blockquote type="cite"><div><blockquote type="cite">2. Given that the client is not closing connections cleanly ( I am looking into this ) but rabbit is successfully detecting that the connection HAS closed, why would we get an out of file descriptors error? I would hope that file descriptors are released properly and re-used<br></blockquote><br>Should be. lsof or rumaging around in /proc/$PID/fd will show you what<br>is open and what isn't. You may be able to detect a leak there. Also<br>netstat -anp | grep beam may be illuminatory.<br><br><blockquote type="cite">This server was running for a few months before this issue cropped up so it may be a 'slow' bug and is not super critical for us as a restart fixed things. However hopefully someone can shed some light<br></blockquote><br>Sure. Whilst I'm sure I sound like a stuck record these days, this bug<br>is fixed in the new persister branch, where we track file descriptors<br>including sockets and stop accepting new network connections when we get<br>near the limit. The goal here is really to protect the erlang VM so that<br>we don't die horribly when we run out of file descriptors. My recent<br>blog post to the lshift blog explains in some detail how the general<br>mechanism works.<br><br><a href="http://www.lshift.net/blog/2010/03/23/the-fine-art-of-holding-a-file-descriptor">http://www.lshift.net/blog/2010/03/23/the-fine-art-of-holding-a-file-descriptor</a><br><br></div></blockquote><div><br></div><div>Thanks for the info, I have seen this response from you about quite a few issues ;) but that's the way things go</div><div><br></div><div>Good blog post that I hadn't seen yet</div><br><blockquote type="cite"><div>Matthew<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Brendan Doyle <br>Manager, Application Development <br>Epic Advertising - New York, Toronto, San Francisco, London <br><a href="http://www.EpicAdvertising.com">www.EpicAdvertising.com</a> <br>60 Columbia Way, Suite 310 <br>Markham, ON L3R 0C9 <br>(905) 946-0300 x.2358 work </div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">(647) 885-7159 mobile<br>(888) 666-3120 fax <br><a href="mailto:brendan.doyle@epicadvertising.com">brendan.doyle@epicadvertising.com</a></div></span></span>
</div>
<br></div></div></body></html>