Hi Jerry,<div><br></div><div>I neglected to mentioned that I am not hitting the memory-based flow control according to my memory alarm stat from�</div><div>rabbitmqadmin.py list nodes</div><div><br></div><div>What threshold would the TCP back pressure logic hit? I assume when the memory limit is reached rabbit pushes back on the publishers? Rabbit did not use more then 2G of RAM out of the 5G allowed for it, if that is the case.</div>
<div><br></div><div>Also is there a way to tell rabbit drop the messages instead of block?</div><div><br><br><div class="gmail_quote">On Tue, Jan 31, 2012 at 12:56 PM, Jerry Kuch <span dir="ltr"><<a href="mailto:jerryk@vmware.com">jerryk@vmware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, Dathan...<br>
<br>
What are your consumers doing with the published messages? �If you use rabbitmqctl or<br>
the management plugin to look at what's going on in your queues, do you see messages<br>
accumulating but not being delivered? �Or delivered but not ACKed? �If messages are<br>
building up (either undelivered or unACKed) faster than consumers are draining them,<br>
you might be hitting memory-based flow control, which will use TCP back pressure to<br>
stop the publishers.<br>
<br>
See here for more information:<br>
<br>
<a href="http://www.rabbitmq.com/memory.html" target="_blank">http://www.rabbitmq.com/memory.html</a><br>
<br>
To get an idea whether this is happening to you, check out your queue contents as<br>
suggested above, and see if memory alarms are being set in your rabbit logs.../<br>
<br>
Best regards,<br>
Jerry<br>
<div class="im"><br>
----- Original Message -----<br>
From: "Dathan Pattishall" <<a href="mailto:dathan@schoolfeed.com">dathan@schoolfeed.com</a>><br>
To: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
Sent: Tuesday, January 31, 2012 12:52:55 PM<br>
Subject: [rabbitmq-discuss] Problems with rabbit and hoping I can get some � � �help<br>
<br>
<br>
<br>
<br>
Let me first describe my setup.<br>
<br>
root@webnode1]# rabbitmqadmin.py show overview<br>
+--------------------+-----------------+--------------------+------------------+<br>
| management_version | node | statistics_db_node | statistics_level |<br>
+--------------------+-----------------+--------------------+------------------+<br>
| 2.7.1 | rabbit@webnode1 | rabbit@webnode1 | fine |<br>
+--------------------+-----------------+--------------------+------------------+<br>
<br>
<br>
<br>
</div>Rabbit MQ's producers comes from PHP 5.3.8 <a href="http://www.php.net/manual/en/book.amqp.php" target="_blank">http://www.php.net/manual/en/book.amqp.php</a> . Each apache process could produce a rabbit message, I am producing around 1000 messages a second on c1.xtralarge instance at ec2.<br>
<div class="im"><br>
<br>
My erlang version is<br>
<br>
<br>
/usr/local/bin/erl -v<br>
Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]<br>
<br>
<br>
<br>
<br>
The PROBLEM:<br>
<br>
<br>
After about 40 mins of rabbit accepting messages all connections block causing a rather bad error on the front ends killing traffic. Turning rabbit off and restarting the web servers forces a recovery.<br>
<br>
<br>
Stats from Rabbit:<br>
<br>
<br>
Roughly 5000 queues are made<br>
Roughly 3600 exchanges are made<br>
Each exchange can have at most 1200 queues bound to it.<br>
Each Queue is setup for autodelete and so is the exchanges with delivery type 1.<br>
All data passed is JSON<br>
<br>
<br>
The consumer is NODE and its keeping up with the consumption<br>
<br>
<br>
RabbitMQ memlimit is around 5.3G<br>
RabbitMQ mem used hits around 1.9G when it freezes produces<br>
RabbitMQ proc used hits around 220K<br>
RabbitMQ fd_total is 50K<br>
RabbitMQ socks_total is around 45K and Socks used is 4K<br>
mem_ets hists 100M // not sure what this is<br>
<br>
<br>
Any idea what is going on? What limit am I hitting? Why does RabbitMQ block? How can I detect that I am about to hit a block state? Any suggestions or request of additional data would be great.<br>
<br>
<br>
<br>
<br>
<br>
<br>
</div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br></div>