<div dir="ltr"><div>Thanks Matthias,<br></div><div style><br></div><div style>Knowing that the tokyo plugin is not often used is useful info. I've spent the weekend reading as much as I can and some of the blogs on the rabbit site [e.g. 1,2 were particularly useful]. I don't claim to get all this, but I think I am beginning to get a general understanding and realising we do indeed need a buffer to decouple those consumers that we know are inherently slow or will be taken offline periodically (e.g. the map tile renderers which are maintaining Billions of tiles in HBase in large queue-draining batch operations).</div>
<div style><br></div><div style>Really appreciate the helpful pointers from both you and Jerry - cheers guys,</div><div style>Tim</div><div class="gmail_extra"><br></div><div class="gmail_extra">[1] <a href="http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/">http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/</a></div>
<div class="gmail_extra">[2] <a href="http://www.rabbitmq.com/blog/2011/10/27/performance-of-queues-when-less-is-more/">http://www.rabbitmq.com/blog/2011/10/27/performance-of-queues-when-less-is-more/</a></div><div class="gmail_extra">
<br><div class="gmail_quote">On Sun, Mar 10, 2013 at 2:17 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Tim,<div class="im"><br>
<br>
On 10/03/13 12:14, Tim Robertson wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
it is really not good practice to<br>
deliberately use RMQ as a large buffering technology for queues, due to<br>
the memory management. Namely, that if one queue is hugely backed up,<br>
we'll get into a oscillation of:<br>
i) memory limit hit<br>
ii) block everything while flush partially to disk<br>
iii) repeat immediately (while the disabled consumer remains)<br>
While it will work, we'll likely cripple other parts of the system if<br>
they are going through the same broker.<br>
</blockquote>
<br></div>
Rabbit will start paging to disk *before* the memory limit is hit, in order to prevent precisely the stop-start scenario you describe above.<br>
<br>
Obviously if the publishing happens at a faster rate than rabbit can write to disk then eventually the memory limit will still be hit and producers blocked while rabbit catches up.<br>
<br>
Furthermore, as Jerry says, messages do have a non-zero memory footprint even when paged to disk. As a consequence, when "filling" rabbit with messages, the paging becomes more and more aggressive over time to the point where all memory is taken up by the residual footprint of paged messages, and publishing even a small number of messages will trigger a memory alarm. And eventually *all* memory is consumed that way and publishers are blocked indefinitely until some messages are removed from the queues.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I think this was probably a lack of understanding on our part, as we<br>
anticipated using it as a queue (to do large buffering) whereas I<br>
presume it is (?) really intended to be a messaging system and targeting<br>
zero queue sizes is the expected behavior (consumer throughput matched<br>
to producer).<br>
</blockquote>
<br></div>
We used to say that "an empty rabbit is a healthy rabbit", but that statement predates many improvements to persistence and flow-control that have happened since. So nowadays I'd rephrase that as "a non-obese rabbit is a healthy rabbit".<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Are there alternative configurations that you are aware of that would<br>
allow it to back up large queues, without hitting memory limits? (the<br>
tokyo cabinet plugin perhaps?)<br>
</blockquote>
<br></div>
Exactly that. With the tokyo cabinet plugin, messages that have been paged to disk leave zero memory footprint, allowing rabbit to hold on to as much message data as will fit on disk.<br>
<br>
Note however that the plugin is rarely used and is not included in our distributions due to the dependency on non-Erlang, platform-specific code.<br>
<br>
Alternatively, increase the amount of memory on your rabbit machine.<br>
<br>
Regards,<br>
<br>
Matthias.<br>
</blockquote></div><br></div></div>