<br><br>Let me first describe my setup.<br><br>root@webnode1]# rabbitmqadmin.py show overview<br><font face="'courier new', monospace">+--------------------+-----------------+--------------------+------------------+<br>
| management_version | node | statistics_db_node | statistics_level |<br>+--------------------+-----------------+--------------------+------------------+<br>| 2.7.1 | rabbit@webnode1 | rabbit@webnode1 | fine |<br>
+--------------------+-----------------+--------------------+------------------+</font><br><div><font face="'courier new', monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">Rabbit MQ's producers comes from PHP 5.3.8 <a href="http://www.php.net/manual/en/book.amqp.php">http://www.php.net/manual/en/book.amqp.php</a>. Each apache process could produce a rabbit message, I am producing around 1000 messages a second on c1.xtralarge instance at ec2.</font></div>
<div><span style="font-family:arial,helvetica,sans-serif"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif">My erlang version is </span></div><div><span style="font-family:arial,helvetica,sans-serif"> </span></div>
<div><font face="arial, helvetica, sans-serif"><div>/usr/local/bin/erl -v</div><div>Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]</div><div><br></div><div><br></div><div>The PROBLEM:</div>
<div><br></div><div>After about 40 mins of rabbit accepting messages all connections block causing a rather bad error on the front ends killing traffic. Turning rabbit off and restarting the web servers forces a recovery.</div>
<div><br></div><div>Stats from Rabbit:</div><div><br></div><div>Roughly 5000 queues are made</div><div>Roughly 3600 exchanges are made</div><div>Each exchange can have at most 1200 queues bound to it.</div><div>Each Queue is setup for autodelete and so is the exchanges with delivery type 1.</div>
<div>All data passed is JSON</div><div><br></div><div>The consumer is NODE and its keeping up with the consumption</div><div><br></div><div>RabbitMQ memlimit is around 5.3G</div><div>RabbitMQ mem used hits around 1.9G when it freezes produces</div>
<div>RabbitMQ proc used hits around 220K</div><div>RabbitMQ fd_total is 50K</div><div>RabbitMQ socks_total is around 45K and Socks used is 4K</div><div>mem_ets hists 100M // not sure what this is </div><div><br></div><div>
Any idea what is going on? What limit am I hitting? Why does RabbitMQ block? How can I detect that I am about to hit a block state? Any suggestions or request of additional data would be great.</div><div><br></div><div><br>
</div><div><br></div></font></div>