<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:275717115;
mso-list-type:hybrid;
mso-list-template-ids:-1097149176 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hello folks, we’re trying to troubleshoot our MQ clusters that kept partitioning, despite their use of a direct cross-over connection to avoid issues with a switch failing or cycling. We have 6 servers split into 3 clusters. All boxes accept traffic from producers and consumers on eth0 (connected to the switch) and eth1 is connected the other box in the clustered pair. We use a host file override on each box to direct MQ traffic over the crossover and Rabbit binds to all IPs. Both NICs on each box are 1gs.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Despite the cross-over we were seeing network partition alerts with version 3.2.2. We saw NIC reset errors (Intel NICs) and just upgraded the drivers to fend off that problem and tried some buffer tuning. But we’re still dropping packets on the cross-over interface so I’m worried the partitions may continue. Here are the questions I have:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>1)<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Is it a bad idea to use a cross-over like this? <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>2)<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>We’re seeing ~2.5Mbps in / ~10Mbps out on the public eth0 interface but ~45Mbps in / ~30Mbps out on the cross-over. Is that kind of amplification normal?<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>3)<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>If it’s ok to use the cross-over, what TCP tuning am I missing?<o:p></o:p></p><p class=MsoListParagraph><o:p> </o:p></p><p class=MsoNormal>Here are some more stats from our setup:<o:p></o:p></p><p class=MsoNormal>Ubuntu 12.04 3.2.0-30-generic<o:p></o:p></p><p class=MsoNormal>~5000 connections / 6000 queues / 12000 channels per cluster<o:p></o:p></p><p class=MsoNormal>~1 dropped packet ever few minutes on the cross-over if. No errors or overruns, etc<o:p></o:p></p><p class=MsoNormal>net.ipv4.tcp_wmem = 10240 524288 16777216 <o:p></o:p></p><p class=MsoNormal>net.ipv4.tcp_rmem = 10240 524288 16777216 <o:p></o:p></p><p class=MsoNormal>net.core.rmem_max = 16777216 <o:p></o:p></p><p class=MsoNormal>net.core.wmem_max = 16777216<o:p></o:p></p><p class=MsoNormal>and the “packets collapsed in receive queue due to low socket buffer” keeps incrementing<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>