<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Jerry,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="border:none;padding:0in"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> rabbitmq-discuss-bounces@lists.rabbitmq.com [mailto:rabbitmq-discuss-bounces@lists.rabbitmq.com]
<b>On Behalf Of </b>Jerry Kuch<br>
<b>Sent:</b> Wednesday, January 16, 2013 4:02 PM<br>
<b>To:</b> Discussions about RabbitMQ<br>
<b>Subject:</b> Re: [rabbitmq-discuss] Rabbitmq boot failure with "tables_not_present"<o:p></o:p></span></p>
</div>
<div style="mso-element:para-border-div;border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">On Wed, Jan 16, 2013 at 11:54 AM, Zhao, Shanyu <<a href="mailto:shanyu.zhao@intel.com" target="_blank">shanyu.zhao@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<div>
<div>
<div style="mso-element:para-border-div;border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="border:none;padding:0in"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div style="mso-element:para-border-div;border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
The relevant part of the log is shown below. But the problem is that we saw these log messages repeated every 7-8 seconds and can last as long as 80 minutes before rabbit finally start up correctly. During this time any connection to the rabbitmq cluster will
get a disconnected exception.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
Any idea on what might have caused this problem? <o:p></o:p></p>
</div>
</div>
</blockquote>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div style="mso-element:para-border-div;border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
=INFO REPORT==== 16-Jan-2013::14:11:37 ===<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
Error description:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{case_clause,{error,tables_not_present}}<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
Log files (may contain more information):<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
<a href="mailto:/var/log/rabbitmq/rabbit@ip-10-0-2-97.log">/var/log/rabbitmq/rabbit@ip-10-0-2-97.log</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
<a href="mailto:/var/log/rabbitmq/rabbit@ip-10-0-2-97-sasl.log">/var/log/rabbitmq/rabbit@ip-10-0-2-97-sasl.log</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
Stack trace:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
[{rabbit_mnesia,discover_cluster,1},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{rabbit_mnesia,init_from_config,0},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{rabbit_mnesia,init,0},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{rabbit,'-run_boot_step/1-lc$^1/1-1-',1},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{rabbit,run_boot_step,1},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{rabbit,'-start/2-lc$^0/1-0-',1},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{rabbit,start,2},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{application_master,start_it_old,4}]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
=INFO REPORT==== 16-Jan-2013::14:11:38 ===<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
application: rabbit<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
exited: {bad_return,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{{rabbit,start,[normal,[]]},<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{'EXIT',<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{rabbit,failure_during_boot,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0in">
{case_clause,{error,tables_not_present}}}}}}<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:9.6pt;border:none;padding:0in">
type: temporary<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<div style="mso-element:para-border-div;border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
</div>
<div>
<div style="mso-element:para-border-div;border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="border:none;padding:0in">You mention that you sometime see this after a redeploy. Depending on how you've redeployed, have you successfully clustered the nodes in the first place? The error means that some of the tables in Erlang's
Mnesia distributed database upon which Rabbit relies to maintain broker metadata weren't found, suggesting that some prior state or configuration perished during your redeploy process.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think during the time the error logs are generated, the cluster may not be successfully formed. As part of the deployment scripts, I deleted all content in
/var/lib/rabbitmq/mnesia to recover from some scenario when cluster cannot be formed. Here is the relevant part of the deployment scripts:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">sudo("bash -c 'echo XXXXXXXXXXXXXXXX > /var/lib/rabbitmq/.erlang.cookie'")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">sudo("chown rabbitmq /var/lib/rabbitmq/.erlang.cookie")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">sudo("chmod 600 /var/lib/rabbitmq/.erlang.cookie")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">sudo("rm -fr /var/lib/rabbitmq/mnesia")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">What I want to achieve after redeployment is to erase previous states completely and let the cluster starts with a clean state, that’s why I erased the /mnesia
folder (is there a better way to do that?). The problem is sometimes the error messages show up for a few minutes then everything works fine after that, but other times I saw the error message being logged for 80 minutes before the cluster works correctly.
Do you have any suggestions?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Shanyu<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>