<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Segoe UI","sans-serif";
color:black;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Segoe UI","sans-serif";
color:#17365D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D">Even after upping the net_tick_time to 120 we still have had issues with the cluster partioning. Although today we came up with evidence that this is tied
to our deployment process to our production systems. Hence I am back to thinking this is a network or I/O problem.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D">Is there anything we can watch specifically when we do the deploy to catch what is going on in the erlang / rabbit world? As I mentioned before we are not
seeing anything overt in our monitoring or performance charts. We plan on re-looking at it as we re-enact the deployment over and over again. Our focus will be pretty much on the network and NIC I/O and virtual interfaces, etc. Previous checks of the deployment
stage showed overall network usage spiked to about 30% so it seemed unlikely, but….<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D">Can someone confirm for me that I understand these events correctly? I would really appreciate it<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">=INFO REPORT==== 9-Dec-2013::05:43:41 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">rabbit on node 'rabbit@NURWEB-QAAPP01' down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">// The above indicates the net_ticktime expired without a good response?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">=ERROR REPORT==== 9-Dec-2013::05:43:41 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">Mnesia('rabbit@NURWEB-QAWEB01'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@NURWEB-QAAPP01'}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">// This means it got back in touch with QAAPP01 and neither had gotten good results from the net_ticktime specified “pings”. So it partitioned itself off. So close
to it being down though?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">=ERROR REPORT==== 9-Dec-2013::05:43:41 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">** Generic server <0.323.0> terminating<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">** Last message in was {mnesia_locker,'rabbit@NURWEB-QAAPP01',granted}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">** When Server state == {state,<0.321.0>,<0.322.0>,rabbit_mgmt_sup,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"> [{rabbit_mgmt_db,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"> {rabbit_mgmt_db,start_link,[]},<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"> permanent,4294967295,worker,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"> [rabbit_mgmt_db]}]}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">** Reason for termination ==
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">** {unexpected_info,{mnesia_locker,'rabbit@NURWEB-QAAPP01',granted}}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">// This (above) I have seen rarely, but it seems related – any ideas aside from the node crashed?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">=INFO REPORT==== 9-Dec-2013::05:43:43 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">only running disc node went down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">// Above indicates the cluster as it is now no longer has QAAPP01 which is the disk node.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">=ERROR REPORT==== 9-Dec-2013::05:45:35 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New";color:#17365D">Mnesia('rabbit@NURWEB-QAWEB01'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@NURWEB-QAWEB02'}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D">// The above indicates that it just portioned itself off from another of the cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D">Any help would be very much appreciated – especially confirming that I understand the above events will help with the troubleshooting process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:gray"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-family:"Arial","sans-serif";color:#F0982E">Bill Chmura
</span></b><span style="color:#17365D"> </span><i><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#7C7C7C">Director, IT Development Services</span></i><span style="color:#17365D"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:#17365D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Bill Chmura
<br>
<b>Sent:</b> Tuesday, December 03, 2013 4:54 PM<br>
<b>To:</b> 'rabbitmq-discuss@lists.rabbitmq.com'<br>
<b>Subject:</b> Recurring partitioning problem on local network <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">We are experiencing a frequent partitioning problem on our network with our RabbitMQ cluster. We’ve not been able to iron it out and are running out of time
before this needs to move into production.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">I’ll just focus on our DEV environment as it is just a scaled down version of the other environments.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Running on ESXi virtual machines (4GB Ram, 4 Cores dedicated)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Windows2008 R2 SP1 64-Bit<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">RabbitMQ 3.2.0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Erlang R16B02<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">There are two web servers and one app server in the above config. These are all connected to the same network. Each servers is running its own node – clustering
is done through the rabbit config file.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Devweb01 – Ram<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Devweb02 – Ram<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">DevApp01 - Disk<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">The problem is that every once in a while it starts partitioning off nodes, with nothing really correlating with it happening… not big traffic on the network,
no disruptions we can find, etc. We have gone through and made sure there were no VM settings that allowed items to “go to sleep” or anything aside from a “high performance setting” (versus power savings).
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Here is what we are seeing in the logs… which to me looks like a network interruption, but nothing else indicated that the machine was having issues. We have
a load balancer that flags systems with problems, we have a zenoss node monitoring the servers, we checked the ESXi charts and logs, we looked through windows system logs… nothing seems to have been amiss.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">In one partitioning event we saw this in the WebApp server log:<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=INFO REPORT==== 27-Nov-2013::18:11:07 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">rabbit on node 'rabbit@NURWEB-DEVWEB01' down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=ERROR REPORT==== 27-Nov-2013::18:11:10 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Mnesia('rabbit@NURWEB-DEVAPP01'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@NURWEB-DEVWEB01'}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">And in the DevWeb01 machine mentioned’s logs we saw this – it also mentions it lost connections to both of the other boxes.<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=INFO REPORT==== 27-Nov-2013::18:10:53 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">rabbit on node 'rabbit@NURWEB-DEVAPP01' down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=ERROR REPORT==== 27-Nov-2013::18:10:53 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Mnesia('rabbit@NURWEB-DEVWEB01'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@NURWEB-DEVAPP01'}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=ERROR REPORT==== 27-Nov-2013::18:10:59 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Mnesia('rabbit@NURWEB-DEVWEB01'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@NURWEB-DEVWEB02'}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=INFO REPORT==== 27-Nov-2013::18:11:00 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">only running disc node went down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=INFO REPORT==== 27-Nov-2013::18:11:01 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">rabbit on node 'rabbit@NURWEB-DEVWEB02' down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=INFO REPORT==== 27-Nov-2013::18:11:04 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">only running disc node went down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">And web02 only mentions the web01 going down<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=INFO REPORT==== 27-Nov-2013::18:11:09 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">rabbit on node 'rabbit@NURWEB-DEVWEB01' down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">=ERROR REPORT==== 27-Nov-2013::18:11:11 ===<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Mnesia('rabbit@NURWEB-DEVWEB02'): ** ERROR ** mnes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">None of the rabbit installs are actually down during this…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">We’ve seen the same thing on our qa and production boxes – which are the same configurations, just with more nodes. Not many though 8 nodes on production.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black">Any ideas would be really appreciated! I’ve recently added a net_ticktime to my dev servers to set that at 120 (double I believe) to see if that helps.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-family:"Arial","sans-serif";color:#F0982E">Bill
</span></b><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><br>
This email and all attachments are confidential and intended solely <br>
for the use of the individual or entity to which they are addressed. <br>
If you have received this email in error please notify the sender <br>
by replying to this message. If you are not the intended recipient, <br>
please delete this message and all attachments immediately. Do not <br>
copy, disclose, use or act upon the information contained. Please <br>
note that any views or opinions presented in this email are solely <br>
those of the author and do not necessarily represent those of the <br>
company. Finally, the recipient should check this email and any <br>
attachments for the presence of viruses. While every attempt is made <br>
to verify that the contents are safe, the company accepts no liability <br>
for any damage caused by any virus transmitted by this email.</p></body>
</html>