On Wed, Jan 16, 2013 at 11:54 AM, Zhao, Shanyu <span dir="ltr"><<a href="mailto:shanyu.zhao@intel.com" target="_blank">shanyu.zhao@intel.com</a>></span> wrote:<br><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><p class="MsoNormal">The relevant part of the log is shown below. But the problem is that we saw these log messages repeated every 7-8 seconds and can last as long as 80 minutes before rabbit finally start up correctly. During this time any connection to the
rabbitmq cluster will get a disconnected exception.</p><p class="MsoNormal"><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Any idea on what might have caused this problem? </p></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple">
<p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal">=INFO REPORT==== 16-Jan-2013::14:11:37 ===<u></u></p><p class="MsoNormal"><u></u></p>
<p class="MsoNormal">Error description:<u></u><u></u></p>
<p class="MsoNormal"> {case_clause,{error,tables_not_present}}<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Log files (may contain more information):<u></u><u></u></p>
<p class="MsoNormal"> /var/log/rabbitmq/rabbit@ip-10-0-2-97.log<u></u><u></u></p>
<p class="MsoNormal"> /var/log/rabbitmq/rabbit@ip-10-0-2-97-sasl.log<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Stack trace:<u></u><u></u></p>
<p class="MsoNormal"> [{rabbit_mnesia,discover_cluster,1},<u></u><u></u></p>
<p class="MsoNormal"> {rabbit_mnesia,init_from_config,0},<u></u><u></u></p>
<p class="MsoNormal"> {rabbit_mnesia,init,0},<u></u><u></u></p>
<p class="MsoNormal"> {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},<u></u><u></u></p>
<p class="MsoNormal"> {rabbit,run_boot_step,1},<u></u><u></u></p>
<p class="MsoNormal"> {rabbit,'-start/2-lc$^0/1-0-',1},<u></u><u></u></p>
<p class="MsoNormal"> {rabbit,start,2},<u></u><u></u></p>
<p class="MsoNormal"> {application_master,start_it_old,4}]<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">=INFO REPORT==== 16-Jan-2013::14:11:38 ===<u></u><u></u></p>
<p class="MsoNormal"> application: rabbit<u></u><u></u></p>
<p class="MsoNormal"> exited: {bad_return,<u></u><u></u></p>
<p class="MsoNormal"> {{rabbit,start,[normal,[]]},<u></u><u></u></p>
<p class="MsoNormal"> {'EXIT',<u></u><u></u></p>
<p class="MsoNormal"> {rabbit,failure_during_boot,<u></u><u></u></p>
<p class="MsoNormal"> {case_clause,{error,tables_not_present}}}}}}<u></u><u></u></p>
<p class="MsoNormal" style="text-indent:9.6pt">type: temporary</p></div></blockquote><div><br></div><div>You mention that you sometime see this after a redeploy. Depending on how you've redeployed, have you successfully clustered the nodes in the first place? The error means that some of the tables in Erlang's Mnesia distributed database upon which Rabbit relies to maintain broker metadata weren't found, suggesting that some prior state or configuration perished during your redeploy process.<br>
</div><div><br></div><div>Best regards,</div><div>Jerry</div><div><br></div></div>