<div dir="ltr"><div class="gmail_extra">2013/7/19 Kobi Biton <span dir="ltr"><<a href="mailto:kbiton@outbrain.com" target="_blank">kbiton@outbrain.com</a>></span><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div> this is to our understanding </div><div>in order to protect the broker and the consumer as the producers are faster then the consumer ?</div></blockquote><div><br></div><div>Yes, because messages have certain RAM cost.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div></div><div>The problem is that at some point (when the READY messages count is high ~ 1M) rabbitmq blocks most of our producers and does not seem to release them , until we restart them (logstash daemon) on every one of them , we tried purging the queue / restarting rabbitmq , only restarting the producers seems to bring things to normal state.</div>
</blockquote><div><br></div><div>It's driven by memory watermark and/or available disk space. See</div><div><br></div><div><a href="http://www.rabbitmq.com/memory.html">http://www.rabbitmq.com/memory.html</a><br></div>
<div><br></div><div>RabbitMQ log should make it clear what limit (RAM or disk) was reached.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br></div><div>I guess my questions are:</div><div><br></div><div> - Is my problem on the consumer side ? I am unable to debug the consumer speed or state</div></blockquote><div><br></div><div>Yes, consumers do not keep up. Try with a larger number of them.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div> - Can I tune rabbitmq for lots of connections and high message rate?</div>
</blockquote><div><br></div><div>It's not really about the number of connections or message rates. See <a href="http://www.rabbitmq.com/memory.html">http://www.rabbitmq.com/memory.html</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div> - We use fanout exchange , when a consumer creates a new queue under this exchange and does not consume fast enough can he effect the producers from the other queue (i.e cause them to be blocked?)</div></blockquote>
</div><br>All connections that attempt to publish anything when one of the limits is hit</div><div class="gmail_extra">will be throttled.</div><div class="gmail_extra"><br></div><div class="gmail_extra">You can reduce RAM footprint of individual messages by using <a href="https://github.com/rabbitmq/rabbitmq-toke">https://github.com/rabbitmq/rabbitmq-toke</a></div>
<div class="gmail_extra">which stores message index in Tokyo Cabinet. Yes, it does not even have a README :(</div><div class="gmail_extra">-- <br>MK<br><br><a href="http://github.com/michaelklishin" target="_blank">http://github.com/michaelklishin</a><br>
<a href="http://twitter.com/michaelklishin" target="_blank">http://twitter.com/michaelklishin</a><br>
</div></div>