<div dir="ltr"><div>Interesting discussion. � I am new to RabbitMQ and I am trying to understand the coding and architecture requirements on the Producer side.<br></div><div><br></div><div>What happens if the Producer cannot push to a queue? �Does that mean that the Producer system should plan for its own local queue until the broker is back? �What do you do in production when you do not want to loose these messages? �</div>
<div><br></div><div><br></div><div>Thanks,</div><div>Thierry</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 18, 2013 at 4:19 PM, Michael Klishin <span dir="ltr"><<a href="mailto:michael.s.klishin@gmail.com" target="_blank">michael.s.klishin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">2013/7/19 Kobi Biton <span dir="ltr"><<a href="mailto:kbiton@outbrain.com" target="_blank">kbiton@outbrain.com</a>></span><br>
<div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div> this is to our understanding�</div><div>in order to protect the broker and the consumer as the producers are faster then the consumer ?</div></blockquote><div><br></div></div><div>Yes, because messages have certain RAM cost.</div>
<div class="im">
<div>�</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div></div><div>The problem is that at some point (when the READY messages count is high ~ 1M) �rabbitmq blocks most of our producers and does not seem to release them , until we restart them (logstash daemon) on every one of them , we tried purging the queue / restarting rabbitmq , only restarting the producers seems to bring things to normal state.</div>
</blockquote><div><br></div></div><div>It's driven by memory watermark and/or available disk space. See</div><div><br></div><div><a href="http://www.rabbitmq.com/memory.html" target="_blank">http://www.rabbitmq.com/memory.html</a><br>
</div>
<div><br></div><div>RabbitMQ log should make it clear what limit (RAM or disk) was reached.</div><div class="im"><div>�</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br></div><div>I guess my questions are:</div><div><br></div><div>�- Is my problem on the consumer side ? I am unable to debug the consumer speed or state</div></blockquote><div><br></div></div><div>Yes, consumers do not keep up. Try with a larger number of them.</div>
<div class="im">
<div>�</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>�- Can I tune rabbitmq for lots of connections and high message rate?</div>
</blockquote><div><br></div></div><div>It's not really about the number of connections or message rates. See�<a href="http://www.rabbitmq.com/memory.html" target="_blank">http://www.rabbitmq.com/memory.html</a></div>
<div class="im"><div>�</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>�- We use fanout exchange , when a consumer creates a new queue under this exchange and does not consume fast enough can he effect the producers from the other queue (i.e cause them to be blocked?)</div></blockquote>
</div></div><br>All connections that attempt to publish anything when one of the limits is hit</div><div class="gmail_extra">will be throttled.</div><div class="gmail_extra"><br></div><div class="gmail_extra">You can reduce RAM footprint of individual messages by using�<a href="https://github.com/rabbitmq/rabbitmq-toke" target="_blank">https://github.com/rabbitmq/rabbitmq-toke</a></div>
<div class="gmail_extra">which stores message index in Tokyo Cabinet. Yes, it does not even have a README :(</div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra">-- <br>MK<br><br><a href="http://github.com/michaelklishin" target="_blank">http://github.com/michaelklishin</a><br>
<a href="http://twitter.com/michaelklishin" target="_blank">http://twitter.com/michaelklishin</a><br>
</div></font></span></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>