<div dir="ltr"><div>Interesting discussion. I am new to RabbitMQ and I am trying to understand the coding and architecture requirements on the Producer side.<br></div><div><br></div><div>What happens if the Producer cannot push to a queue? Does that mean that the Producer system should plan for its own local queue until the broker is back? What do you do in production when you do not want to loose these messages? </div>
<div><br></div><div><br></div><div>Thanks,</div><div>Thierry</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 18, 2013 at 4:19 PM, Michael Klishin <span dir="ltr"><<a href="mailto:michael.s.klishin@gmail.com" target="_blank">michael.s.klishin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">2013/7/19 Kobi Biton <span dir="ltr"><<a href="mailto:kbiton@outbrain.com" target="_blank">kbiton@outbrain.com</a>></span><br>
<div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div> this is to our understanding </div><div>in order to protect the broker and the consumer as the producers are faster then the consumer ?</div></blockquote><div><br></div></div><div>Yes, because messages have certain RAM cost.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div></div><div>The problem is that at some point (when the READY messages count is high ~ 1M) rabbitmq blocks most of our producers and does not seem to release them , until we restart them (logstash daemon) on every one of them , we tried purging the queue / restarting rabbitmq , only restarting the producers seems to bring things to normal state.</div>
</blockquote><div><br></div></div><div>It's driven by memory watermark and/or available disk space. See</div><div><br></div><div><a href="http://www.rabbitmq.com/memory.html" target="_blank">http://www.rabbitmq.com/memory.html</a><br>
</div>
<div><br></div><div>RabbitMQ log should make it clear what limit (RAM or disk) was reached.</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br></div><div>I guess my questions are:</div><div><br></div><div> - Is my problem on the consumer side ? I am unable to debug the consumer speed or state</div></blockquote><div><br></div></div><div>Yes, consumers do not keep up. Try with a larger number of them.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div> - Can I tune rabbitmq for lots of connections and high message rate?</div>
</blockquote><div><br></div></div><div>It's not really about the number of connections or message rates. See <a href="http://www.rabbitmq.com/memory.html" target="_blank">http://www.rabbitmq.com/memory.html</a></div>
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div> - We use fanout exchange , when a consumer creates a new queue under this exchange and does not consume fast enough can he effect the producers from the other queue (i.e cause them to be blocked?)</div></blockquote>
</div></div><br>All connections that attempt to publish anything when one of the limits is hit</div><div class="gmail_extra">will be throttled.</div><div class="gmail_extra"><br></div><div class="gmail_extra">You can reduce RAM footprint of individual messages by using <a href="https://github.com/rabbitmq/rabbitmq-toke" target="_blank">https://github.com/rabbitmq/rabbitmq-toke</a></div>
<div class="gmail_extra">which stores message index in Tokyo Cabinet. Yes, it does not even have a README :(</div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra">-- <br>MK<br><br><a href="http://github.com/michaelklishin" target="_blank">http://github.com/michaelklishin</a><br>
<a href="http://twitter.com/michaelklishin" target="_blank">http://twitter.com/michaelklishin</a><br>
</div></font></span></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>