<div dir="ltr">I upgraded from 3.0.4 to 3.1.5 and from esl-erlang R15B03 to esl-erlang R16B01. I've been doing some digging and can't find any reason why this would happen, no additional log files, nothing. Note there are three nodes involved in a cluster. I upgraded my "dvlp" sample box which wasn't clustered with no problem using the exact same script I used to upgrade my alpha cluster. Below is the script I'm using (might be useful for others upgrading a cluster). I'm going to try recreating my "Alpha" environment and redoing the upgrade. The ONLY thing that I can think of - I was doing stop/starts as part of training of the cluster about 20 minutes before I did the upgrade to test various concepts, e.g. message loss, master node recovery, etc. etc. At this point, if people haven't seen or heard of this, I'll chalk it up to something funky disk or otherwise until I can try and replicate it. My biggest concern at this point is that because i no longer have the backup's of the mnesia database, replicating the environment won't replicate exactly what was in the database, and so I won't be able to replicate it. When I start deploying to our production environment, I won't make that mistake - I'll shutdown rabbit, back up my whole rabbit_data_directory first :)<div style>
Thanks!<br>Jason</div><div><br></div><div><div># ########################################<br></div><div># 04/22/2013 - JasonMcIntosh - core upgrade script to go through each reported node in a cluster and upgrade it</div>
<div># ########################################</div><div>if [ ! -e /data/rabbitmq ]; then</div><div><span class="" style="white-space:pre"> </span>echo "No rabbit found on $1" > /var/log/rabbit_upgrade.log</div>
<div><span class="" style="white-space:pre"> </span>exit 0</div><div>fi</div><div><br></div><div>export LOG_FILE=/var/log/rabbitmq/upgrade.log</div><div>rm -f $LOG_FILE</div><div><br></div><div>export CLUSTER_STATUS="`rabbitmqctl cluster_status`"</div>
<div>export CLUSTER_STATUS="`echo $CLUSTER_STATUS|tr -d ' \n'`"</div><div>echo "Started at `date`" >> $LOG_FILE</div><div>echo "$CLUSTER_STATUS" >> $LOG_FILE</div><div>echo "" >> $LOG_FILE</div>
<div><br></div><div>getServerFQDN() {</div><div><span class="" style="white-space:pre"> </span>SERVER_FQDN=`echo $1 | awk -F@ '{print $2};'`</div><div><span class="" style="white-space:pre"> </span>FQDN=`nslookup $SERVER_FQDN|grep Name|awk -F\: '{print $2}'|sed 's/ //g'`</div>
<div><span class="" style="white-space:pre"> </span>echo $FQDN</div><div>}</div><div><br></div><div>upgradeRabbitNode() {</div><div><span class="" style="white-space:pre"> </span>SERVER_FQDN=`getServerFQDN $1`</div><div><span class="" style="white-space:pre"> </span>echo "Doing upgrade of $SERVER_FQDN" >> $LOG_FILE</div>
<div><span class="" style="white-space:pre"> </span># The upgrade deploy job actually stops the rabbit server, shouldn't need this, but we'll do it anways</div><div style> # Not exact commands below as I'm using bladelogic internal commands to do these, but the idea should be the same.</div>
<div><span class="" style="white-space:pre"> </span>remote_exec ${SERVER_FQDN} service rabbitmq-server stop >> $LOG_FILE</div><div><span class="" style="white-space:pre"> </span>remote_exec $SERVER_FQDN yum -y erase rabbit* erlang* >> $LOG_FILE</div>
<div> remote_exec $SERVER_FQDN yum -y install rabbitmq-server-3.1.5... >> $LOG_FILE<br></div><div> remote_exec $SERVER_FQDN rabbitmq-plugins enable rabbitmq_management rabbitmq_management_agent rabbitmq_management_visualiser rabbitmq_shovel rabbitmq_shovel_management>> $LOG_FILE<br>
</div><div style> remote_exec $SERVER_FQDN chkconfig rabbitmq-server on</div><div>}</div><div><br></div><div><br></div><div>#Get the cluster nodes and pick the first disk node as the "upgrader" node</div>
<div>export UPGRADER_NODE=`echo "$CLUSTER_STATUS"|awk -F\[ '{print $4}'|awk -F\] '{print $1}'|awk -F, '{print $1}'`</div><div>export UPGRADER_FQDN=`getServerFQDN $UPGRADER_NODE`</div><div>
export NODE_LIST="`echo $CLUSTER_STATUS|awk -F\[ '{sub(/.*running_nodes/,\"\")};1'|awk -F\[ '{print $2}'|awk -F\] '{print $1}'|sed -e 's/,/ /g'`"</div><div><br></div><div>
echo "Node list: $NODE_LIST " >> $LOG_FILE</div><div>echo "Disk node for last upgrade $UPGRADER_NODE">> $LOG_FILE</div><div><br></div><div>if [ "$UPGRADER_NODE" = "" ]; then</div>
<div><span class="" style="white-space:pre"> </span>echo " ** No upgrader node found! EXITING" >> $LOG_FILE</div><div><span class="" style="white-space:pre"> </span>exit -1</div><div>fi</div><div><br></div>
<div><br></div><div>#Shutdown and upgrade all other nodes than the upgrader node</div><div>echo "Doing upgrade of all non upgrade nodes..." >> $LOG_FILE</div><div>for clusterNode in ${NODE_LIST}; do</div><div>
<span class="" style="white-space:pre"> </span>if [ $clusterNode != $UPGRADER_NODE ]; then </div><div><span class="" style="white-space:pre"> </span>upgradeRabbitNode $clusterNode</div><div><span class="" style="white-space:pre"> </span>fi</div>
<div>done</div><div><br></div><div>#Upgrade the upgrader node now.</div><div>echo "Upgrade the core upgrade node ..." >> $LOG_FILE</div><div>upgradeRabbitNode $UPGRADER_NODE</div><div><br></div><div>#NOW start all nodes, starting with the upgrader node.</div>
<div>echo "Starting rabbit on upgrader node..." >> $LOG_FILE</div><div>remote_exec $UPGRADER_FQDN service rabbitmq-server start >> $LOG_FILE</div><div>for clusterNode in ${NODE_LIST}; do</div><div><span class="" style="white-space:pre"> </span>if [ $clusterNode != $UPGRADER_NODE ]; then </div>
<div><span class="" style="white-space:pre"> </span>SERVER_FQDN=`getServerFQDN $clusterNode`</div><div><span class="" style="white-space:pre"> </span>echo "Starting rabbit on NON upgrade nodes..." >> $LOG_FILE</div>
<div><span class="" style="white-space:pre"> </span>remote_exec $SERVER_FQDN service rabbitmq-server start >> $LOG_FILE</div><div><span class="" style="white-space:pre"> </span>fi</div><div>done</div><div><br></div>
<div>#Finally, make sure our HA Policy is applied to all our virtual hosts</div><div><br></div><div>echo "Finished with upgrade..." >> $LOG_FILE</div><div>#Report how we worked out...</div><div>echo "</div>
<div>RESULTS</div><div>"</div><div>cat $LOG_FILE</div><div><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 27, 2013 at 4:58 AM, Emile Joubert <span dir="ltr"><<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">On 23/08/13 23:01, Jason McIntosh wrote:<br>
> A ps auf shows /usr/lib/erlang/erts-5.9.3.1/bin/epmd -daemon as still<br>
> running. SO I'm wondering if that might have an impact.<br>
<br>
</div>Depending on how you upgraded Erlang you may need to stop this process<br>
manually. I'd be surprised if this was the cause of the error though.<br>
<div class="im"><br>
> stop rabbit on server X (upgrader is Z, other node is Y)<br>
> remove all rabbit/erlang RPM's<br>
> Reinstall rabbit software<br>
> Update rabbitmqadmin<br>
> Enable management plugins (just in case)<br>
> Enable auto start.<br>
><br>
> Rinse and repeat on servers Y, then Z and then start bringing them up<br>
> starting with upgrader node. First start Z, then start Y, then start X.<br>
<br>
</div>From which versions did you upgrade?<br>
<div class="im"><br>
> On Fri, Aug 23, 2013 at 4:51 PM, Jason McIntosh wrote:<br>
<br>
> =INFO REPORT==== 23-Aug-2013::15:37:45 ===<br>
> Disk free limit set to 1000MB<br>
<br>
</div>Were there any other log messages in either logfile or console messages<br>
on any nodes in the interval between or near 15:37:45 - 15:37:50?<br>
<div class="im"><br>
> =ERROR REPORT==== 23-Aug-2013::15:37:50 ===<br>
> ** Generic server <0.303.0> terminating<br>
> ** Last message in was {'EXIT',<0.350.0>,normal}<br>
</div>Did you perform the same upgrade in other environments, and the failure<br>
only occurred in one of the environments?<br>
<span class=""><font color="#888888"><br>
<br>
<br>
-Emile<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Jason McIntosh<br><a href="http://mcintosh.poetshome.com/blog/">http://mcintosh.poetshome.com/blog/</a><br>573-424-7612
</div></div></div>