<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our Rabbit had a bit of an accident this weekend – the 3.2.1 cluster accidentally got rebooted all at once. When MNESIA went down on the second node, it crashed so hard it dropped the disk. That said, we had the JSON backups on the other
two boxes, neither of which wanted to start without the master, so I decided rather than fight them I’d just pull the backup and rebuild the cluster. When I rebuilt the cluster it came back up on 3.2.4, the latest build, I restored and everything seemed fine.
At first.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">By the next morning the master node of the cluster had 1,020,000 erlang processes running (approximately – the actual number was slightly higher) and was deep in the red. Federation wasn’t working, and so we made the call to cleanly shutdown
the cluster and bring it back. Attempting to run rabbitmqctl on the node with the erlang processes resulted in erl_crash dumps, and it had to be rebooted. Another node took over the master spot, and that was when we noticed the issue.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Federation is spinning up internal connections left, right and center. It had spawn 56,000 connection on the node that failed overnight. We tried to rebuild the cluster, but as soon as we loaded the backup it was the same thing – connections
started spawning, even when it was isolated from the network and any true outside connections. We tried rebuilding it back to 3.2.1, and then 3.2.0, we tried old backups from November and October of last year, we tried restoring backups from our Dev and Stage
environments. All spawn these connection issues.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Finally, we disabled the federation plugin and restarted Rabbit. The connections stopped. We then tried reenabling the plugin and restarting again. In 3.2.1, the connections skyrocket again. In 3.2.4, they climb, but much, much more
slowly – a few dozen connections a minute as opposed to hundreds or thousands.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve attached the output of rabbitmqctl report from one of our nodes while the processes were climbing. We’ve currently disabled federation in prod, but we’re looking for a better solution. Ideas?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you very much,<o:p></o:p></p>
<p class="MsoNormal">Kale Ganann<o:p></o:p></p>
</div>
<p>This communication contains information that is confidential,<br>
proprietary in nature, and may also be attorney-client privileged<br>
and/or work product privileged. It is for the exclusive use of the<br>
intended recipient(s). If you are not the intended recipient(s) or<br>
the person responsible for delivering it to the intended<br>
recipient(s), please note that any form of dissemination,<br>
distribution or copying of this communication is strictly<br>
prohibited and may be unlawful. If you have received this<br>
communication in error, please immediately notify the sender by replying<br>
to this message and delete this email immediately. Thank you for your cooperation. </p>
<p>Please be advised that neither Altegrity, its affiliates, its employees<br>
or agents accept liability for any errors, omissions or damages<br>
caused by delays of receipt or by any virus infection in this<br>
message or its attachments, or which may otherwise arise as a<br>
result of this e-mail transmission.</p></body>
</html>