<html><head></head><body bgcolor="#FFFFFF"><div>Also....</div><div><br></div><div>It is possible for a runaway application to create so many connections and channels that processes and/or memory becomes exhausted. Is it possible this happened in your case? It doesn't sound like it, since your troubles sound like they stated with the partition, but its e good to confirm that. As well as the logs, can you post any rabbitmqctl status or report calls from before/after the problems started?</div><div><br></div><div>BTW, you're not by any chance based in British Colombia are you?</div><div><br></div><div>Cheers,</div><div>Tim Watson</div><div><br>On 16 Oct 2013, at 18:01, Tim Watson <<a href="mailto:watson.timothy@gmail.com">watson.timothy@gmail.com</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div><div><span class="Apple-style-span" style="-webkit-tap-highlight-color: rgba(26, 26, 26, 0.296875); -webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); ">On 16 Oct 2013, at 16:34, David Harrison <<a href="mailto:dave.l.harrison@gmail.com">dave.l.harrison@gmail.com</a>> wrote:</span><br></div><div><br></div><div></div><blockquote type="cite"><div><div dir="ltr">Quick update on the queue count: 56</div></div></blockquote><div><br></div><div>Hmm. That seems perfectly reasonable.</div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 17 October 2013 02:29, David Harrison <span dir="ltr"><<a href="mailto:dave.l.harrison@gmail.com" target="_blank">dave.l.harrison@gmail.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>
<br>
</div>What version of rabbit are you running, and how was it installed?<br></blockquote><div><br></div></div><div>3.1.5, running on Ubuntu Precise, installed via deb package.</div><div class="im"></div></div></div></div></blockquote></div></div></div></div></blockquote><div><br></div><div>Of course - I missed that in the subject line. </div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="im"><div><span class="Apple-style-span" style="-webkit-tap-highlight-color: rgba(26, 26, 26, 0.296875); -webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); ">I think 3.1.5 is the latest stable ??</span><br></div></div></div></div></div></blockquote></div></div></div></div></blockquote><div><br></div><div>Yep.</div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="im"><div><br></div></div><div>I'll take a look, we saw a few "too many processes" messages,</div>
<div>
<br></div></div></div></div></blockquote></div></div></div></div></blockquote><div><br></div><div>That's not a good sign. I can't say we've run into that very frequently - it is possible to raise the limit (on the number of processes), but I suspect that's not the root of this anyway.</div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>"<span>Generic</span> <span>server</span> <span>net_kernel</span> <span>terminating" followed by :</span></div><div><span><br></span></div><div><span><div>
** Reason for termination ==</div><div>** {system_limit,[{erlang,spawn_opt,</div></span></div></div></div></div></blockquote></div></div></div></div></blockquote><div><br></div><div>Yeah - once that goes you're in trouble. That's an unrecoverable error, the equivalent of crashing the jvm.</div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>There was definitely a network partition, but the whole cluster nose dived during the crash</div><div class="im"><div> </div></div></div></div></div></blockquote></div></div></div></div></blockquote><div><br></div><div>Yeah, partitions are bad and can even become unrecoverable without restarts (which is why we warn against using clustering in some environments), but what you're experiencing shouldn't happen.</div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
> The erl_crash.dump slogan error was : eheap_alloc: Cannot allocate 2850821240 bytes of memory (of type "old_heap").<br>
><br>
<br>
</div>That's a plain old OOM failure. Rabbit ought to start deliberately paging messages to disk well before that happens, which might also explain a lot of the slow/unresponsive-ness.<br></blockquote><div><br></div>
</div><div>These hosts aren't running swap, we give them a fair bit of RAM (gave them even more now as part of a possible stop gap) </div><div class="im"><div> </div></div></div></div></div></blockquote></div></div></div></div></blockquote><div><br></div><div>This. I suspect the root of your problem is that you don't have any available swap and somehow ran out I memory. Rabbit should've been paging to disk (by hand, not via swap) once you got within a tolerance level of the high watermark, which is why I'd like to see logs if possible since we might be able to identify what led to runaway process spawns and memory allocation during the partition. My money, for the memory use part, is on error_logger, which has been known to blow up in this way when flooded with large logging terms. During a partition, various things can go wrong leading to crashing processes such as queues, some of which can have massive state that RTS logged leading to potential oom situations like this one. Replacing error logger has been in our radar before, but we've not had strong enough reasons to warrant the expense. If what you've seen can be linked to that however...</div><div><br></div><div>To properly diagnose what you've seen though, I will new to get my hands on those logs. Can we arrange that somehow? </div><br><div>Cheers,</div><div>Tim</div></div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>rabbitmq-discuss mailing list</span><br><span><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a></span><br><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a></span><br></div></blockquote></body></html>