We've put two new rabbit 2.5 hosts into production at the same time as we performed some other major upgrades to our software on 9 August. As designed, our messages queued up in rabbit, and we slowly consumed them. When the queues were empty, rabbit blew up and took out both hosts in the cluster.<div>
<br></div><div>Hosts:</div><div>2 x 12 core AMD Opteron 6172</div><div>8GB ram</div><div><br></div><div>The situation here is that we had lots of requests, 1-2k/sec, coming in on one host, routed to the queues on the second, to which there was connected a pool of consumers. For a variety of reasons, that pool of consumers was smaller than we wanted and we had trouble increasing the count. The first incident occurred after the backlog reported by rabbit reached over 700k; the second peak below 300k. Message size ranged from 2-20k.</div>
<div><br></div><div>In both cases, both rabbit hosts started spooling to disk as free memory dropped. We do not use qos, and so our consumers also had significant memory allocated to frames on their input queues. Eventually, we were able to spawn enough consumers to slowly drain the queues. At the rate messages were arriving, this took awhile, but everything stayed within spec.</div>
<div><br></div><div>Shortly after the queues drained, rabbit became completely unresponsive. Our `watch` process to list_queues and check on the state stopped running. We observed heavy swap churn via kswapd[0-9]+. It appeared that rabbit was trying to load in all of the data that it had paged to disk. I've attached graphs that show when this event occurred, but they're the weekly rollups so it's hidden in the noise. The "max" column is the best record of the limits that were hit, especially the "committed" value which is significantly higher than the 8GB of RAM available.</div>
<div><br></div><div>-Aaron</div><div><br></div>
<div><br>-- <br>Aaron Westendorf<br>Senior Software Engineer<br>Agora Games<br>359 Broadway<br>Troy, NY 12180<br>Phone: <a href="tel:518.268.1000" value="+15182681000" target="_blank">518.268.1000</a><br>
<a href="mailto:aaron@agoragames.com" target="_blank">aaron@agoragames.com</a> <br>
<a href="http://www.agoragames.com" target="_blank">www.agoragames.com</a><br><br>
</div>