<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Hi!</div><div><br></div><div> I've just noticed I had this email unsent, waiting in draft for a few days.</div><div><br></div><div> So this is just for closing down the thread with "feedback".</div>
<div><br></div><div> It seems like, that erlang hangs sometimes, waiting for an empty list of files, with no timeouts... it's probably fixed in version 17.</div><div><br></div><div> Thank you for all your help,</div>
<div>
<br></div><div> Peter</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Yes. The epmd process staying around is not related to anything else you're seeing, that's how epmd is supposed to work.<br>
<br>
epmd is a very simple daemon which just maps names to port numbers for Erlang (beam) processes. It doesn't contain any other state.</blockquote><div>Ok.</div><div><br></div><div> BTW IMO the "unix way" would be that if I start and stop something, the machine should be in the state it was before. No processes left. :) But I can live with this. :) We all like our systems well behaved and deterministic. :)</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Node2 and node3 always exited with the message that they are not the<br>
one stopped the last, and if I removed the locks they tried to connect<br>
to each other - while repairing the mnesia db, and as the network<br>
connect failed (as they were not accepting connections probably because<br>
the db repair was on), they quit, leaving the repair in half. So the<br>
nodes were not able to communicate because of the db repair (this is a<br>
theory only), and as the timeout arrived they quit before the repair<br>
would finish. (It would be nice to have a message like "starting repair"<br>
and "repair finished", as the log message currently is not clear about<br>
the state of the repair, I'm not sure that if the repair ends I'll get a<br>
logitem.)<br>
</blockquote>
<br></div>
If you are talking about the upgrades, then<br></blockquote><div>No. No upgrades. It was telling me, that some db files are damaged, and needs to repair them. I'll copy the messages here, but they don't contain too much information about what's the problem.</div>
<div><br></div><div>...</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
NOTE that, we have the "K20rabbitmq-server stop" running, we have the<br>
rabbitmq processes too, but currently neither rabbitmqctl nor management<br>
plugin is accessible, and the reboot process hung.<br>
</blockquote>
<br></div>
So this is very weird. The beam process is still around, but everything in the shutdown appears to have succeeded.<br>
<br>
By the time "Halting Erlang VM" appears in the logs RabbitMQ is completely stopped, that log message is literally the last thing we do before telling the VM to stop.<div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br></blockquote></div>
No, it really shouldn't.<br>
<br>
The beam process has already unregistered from epmd (hence "rabbit seems not to be running at all" in the status command above).<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The running rabbit processes and their open files are here:<br>
<br>
<a href="http://pastebin.com/mLKf5mEu" target="_blank">http://pastebin.com/mLKf5mEu</a><br>
</blockquote>
<br></div>
And it seems to have few files open, and to have closed all network sockets.<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Any ideas? How can I debug this?<br>
Specific questions?<br>
</blockquote>
<br></div>
I am not full of ideas here, this looks like more of an issue with Erlang than RabbitMQ. But some things to look into:<br>
<br>
The beam process (2236 in your case) is the interesting one; it's the thing that should have shut down but hasn't; everything else is pretty much as expected. Unfortunately although it's an Erlang process it has stopped listening for Erlang distribution messages so debugging it is likely to be hard. Having said that, strace might give some clue as to what it's doing, and the small bright side is that so much within that process has already shut down that *anything* it is still doing is a good candidate for what's making it stuck.<br>
<br>
So what does strace say for it?<div><br></div></blockquote><div><br></div><div>CUT STRAIGHT: This is probably the erlang bug that's been fixed in 17, as you've written in the next email. </div><div> </div>
<div># strace -p 2236</div><div>Process 2236 attached</div><div>select(0, NULL, NULL, NULL, NULL</div><div><br></div><div> that means:</div><div> - zero filedescriptors to look for</div><div> - empty read filedescriptorlist</div>
<div> - empty write filedescriptorlist</div><div> - empty exception filedescriptor list <br></div><div> - empty timeout....</div><div><br></div><div>from select(2):</div><div><br></div><div>"If timeout is NULL (no timeout), select() can block indefinitely."<br>
</div><div><br></div><div>This is the cpu-friendly version of </div><div><br></div><div>while(1);</div><div><br></div><div> :)</div><div><br></div><div> I'll try to unbalance this via sending the process signals sigint, sighup, sigterm, sigkill...</div>
<div><br></div><div>Let's try: SIGINT (2)</div><div><br></div><div>.....</div><div><br></div><div>result:</div><div><br></div><div><div>Connection to ... closed by remote host.</div></div><div><br></div><div>(Note: we we're in a shutdown, when the hangup happened :D)</div>
<div><br></div><div>So, the problem is a "wait for no files with no timeout", which MAY be the same bug, erlang 17 has fixed, but we don't know for sure. :)</div><div><br></div><div><br></div><div>If someone founds a hanging rabbit process after stopping, strace it and check whether its a 'select (0,null,null,null', :)</div>
<div><br></div><div><div>Thanks Simon for your help again!</div></div><div><br></div><div> Bye,</div><div><br></div><div> Peter</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Thank you for your help, I hope these hangups could get eliminated.<br>
</blockquote>
<br></div>
I would hope so. We don't have much to go on though.<div><div><br>
<br>
Cheers, Simon<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</div></div></blockquote></div><br></div></div>