<div dir="ltr">Please Note , We have Both Node Running in Same AZ in EC2, and observed Network Partition...</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, May 28, 2014 at 10:45 PM, Rohit Mukherjee <span dir="ltr"><<a href="mailto:rohitmukherjee26@gmail.com" target="_blank">rohitmukherjee26@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Full Error Report :<div><br></div><div><div>=ERROR REPORT==== 28-May-2014::14:41:36 ===</div><div>** Generic server <0.191.0> terminating</div>
<div>** Last message in was {mnesia_locker,'rabbit@vpc-prod-rabbitmq01',granted}</div>
<div>** When Server state == undefined</div><div>** Reason for termination ==</div><div>** {unexpected_info,{mnesia_locker,'rabbit@vpc-prod-rabbitmq01',granted}}</div><div><br></div><div>=ERROR REPORT==== 28-May-2014::14:41:36 ===</div>
<div>** Generic server <0.194.0> terminating</div><div>** Last message in was {mnesia_locker,'rabbit@vpc-prod-rabbitmq01',granted}</div><div>** When Server state == undefined</div><div>** Reason for termination ==</div>
<div>** {unexpected_info,{mnesia_locker,'rabbit@vpc-prod-rabbitmq01',granted}}</div><div><br></div><div>=ERROR REPORT==== 28-May-2014::14:41:36 ===</div><div>** Generic server <0.192.0> terminating</div><div>
** Last message in was {mnesia_locker,'rabbit@vpc-prod-rabbitmq01',</div><div> {not_granted,</div><div> {cyclic,'rabbit@vpc-prod-rabbitmq01',</div><div> {gm_group,</div>
<div> {resource,<<"/">>,queue,<<"impression_click_es">>}},</div><div> write,write,</div><div> {tid,1215,<7209.30767.61>}}}}</div>
<div>** When Server state == undefined</div><div>** Reason for termination ==</div><div>** {unexpected_info,</div><div> {mnesia_locker,'rabbit@vpc-prod-rabbitmq01',</div><div> {not_granted,</div><div>
{cyclic,'rabbit@vpc-prod-rabbitmq01',</div><div> {gm_group,</div><div> {resource,<<"/">>,queue,<<"impression_click_es">>}},</div>
<div> write,write,</div><div> {tid,1215,<7209.30767.61>}}}}}</div><div><br></div><div>=ERROR REPORT==== 28-May-2014::14:41:36 ===</div><div>** Generic server <0.193.0> terminating</div>
<div>** Last message in was {mnesia_locker,'rabbit@vpc-prod-rabbitmq01',granted}</div><div>** When Server state == undefined</div><div>** Reason for termination ==</div><div>** {unexpected_info,{mnesia_locker,'rabbit@vpc-prod-rabbitmq01',granted}}</div>
<div><br></div></div></div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">On Wed, May 28, 2014 at 10:42 PM, Rohit Mukherjee <span dir="ltr"><<a href="mailto:rohitmukherjee26@gmail.com" target="_blank">rohitmukherjee26@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi ALL,<div><br></div><div>We are seeing node getting down and seeing 3 Similar Error Always..</div><div>
<br></div><div><div>1)</div><div> =ERROR REPORT==== 28-May-2014::14:31:49 ===</div><div>AMQP connection <0.6576.0> (running), channel 0 - error:</div><div>
<div>{amqp_error,connection_forced,</div><div> "broker forced connection closure with reason 'shutdown'",none}</div><div><br></div></div><div>2)</div><div><br></div><div>[10:39:58 PM] rohit mukherjee: Mirrored queue 'req_resp_es' in vhost '/': Slave <rabbit@vpc-prod-rabbitmq02.1.256.0> saw deaths of mirrors <rabbit@vpc-prod-rabbitmq01.2.29560.57></div>
<div><br></div><div><br></div><div> 3)</div><div> =ERROR REPORT==== 28-May-2014::14:31:51 ===</div><div>Mnesia('rabbit@vpc-prod-rabbitmq02'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, 'rabbit@vpc-prod-rabbitmq01'}</div>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">Please Suggest Solution for the same</div><div class="gmail_extra"><br></div><div class="gmail_extra">Regards,</div><div class="gmail_extra">Rohit<div><div>
<br><br><div class="gmail_quote">
On Thu, May 22, 2014 at 6:40 PM, Rohit Mukherjee <span dir="ltr"><<a href="mailto:rohitmukherjee26@gmail.com" target="_blank">rohitmukherjee26@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><p class="MsoNormal">HI Team,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">RabbitMQ nodes are getting shutdown automatically , Can you
please suggest how should I fix the issue ..Please find the Log Strace Below </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 22-May-2014::09:43:12 ===</p>
<p class="MsoNormal">Mirrored queue 'win_es' in vhost '/': Promoting slave
<rabbit@vpc-prod-rabbitmq02.1.10232.0> to master</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">Stopping RabbitMQ</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">stopped TCP Listener on [::]:5672</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10638.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10358.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10524.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10355.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10387.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal"> "broker forced connection closure
with reason 'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10352.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10384.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10463.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10349.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10381.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10493.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=ERROR REPORT==== 22-May-2014::09:58:07 ===</p>
<p class="MsoNormal">AMQP connection <0.10346.0> (running), channel 0 -
error:</p>
<p class="MsoNormal">{amqp_error,connection_forced,</p>
<p class="MsoNormal">
"broker forced connection closure with reason
'shutdown'",none}</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=INFO REPORT==== 22-May-2014::09:58:08 ===</p>
<p class="MsoNormal">Halting Erlang VM</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">=====</p><div>Regards,</div>Rohit Mukherjee<br><br>
</div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div>Regards,</div>Rohit Mukherjee<br><br>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div>Regards,</div>Rohit Mukherjee<br><br>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Regards,</div>Rohit Mukherjee<br><br>
</div>