<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<base href="x-msg://591/"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Georgia;
panose-1:2 4 5 2 5 4 5 2 3 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0mm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Georgia","serif";
color:black;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black">Hi Tim,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black">thanks for answer.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black">I would reckon that 'slowing down' the transfer from disk to memory may be a way to go. It definitely is not desirable for all purposes but for particular cases –
like the one I tested – it would be nice to be able to set some values :) Another effect I've seen was using of huge amount of file descriptors which after all limits the transfer from RMQ to consumer as RMQ has to wait. The delivery was very slow (at least
in comparison with publishing). <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black">I know 1G of memory is not much but I look for the limits for various purposes so this is the way to go in my case.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black">thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black">Marek<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif";color:black"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0mm 0mm 0mm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0mm 0mm 0mm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> rabbitmq-discuss-bounces@lists.rabbitmq.com [mailto:rabbitmq-discuss-bounces@lists.rabbitmq.com]
<b>On Behalf Of </b>Tim Watson<br>
<b>Sent:</b> 03 April, 2013 14:36<br>
<b>To:</b> Discussions about RabbitMQ<br>
<b>Subject:</b> Re: [rabbitmq-discuss] memory consumption during message delivery<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Marek,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On 3 Apr 2013, at 13:16, Cermak, Marek wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif"">Hi everybody,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif"">I did some measurements of memory consumption of RMQ (v3.0.2) during sending and delivering messages and found an interesting behaviour during delivery. In short: messages were
stored in mnesia transient storage on disk and when a consumer starts to consume them all of them were read to memory in one chunk which was almost deadly for RMQ and the server.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The mnesia database is used to store schema elements (queue definitions, exchanges, users, etc), however messages are *never* stored in mnesia, they're stored in a custom file system storage layer - the persister - which was purpose written
for RabbitMQ.<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif"">The scenario was 1+2+8+10 pairs of bound direct exchanges and queues each received messages of particular size (1kB, 64 kB, 1 MB and 8MB) and after a while consumer was started
to 'empty' each of the queues. The consumer of the queue with biggest messages triggered the avalanche of data being transferred from disk to memory. Similar behaviour can be seen always when you fill the queue up to the roof with bigger messages and start
the consumer.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal">[snip]<br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif"">Tested with 1G of memory, 64b Debian GNU/linux, Erlang R15B, RMQ v3.0.2.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif"">Is it an expected behaviour? Can it be somehow tuned/adjusted? Thanks.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Well if you've got a bunch of data on disk that needs to be delivered to a consumer later on, it'll need to be read into memory in order to be transferred onto the wire. One obvious way to avoid getting into this position would be to allocate
a good deal more memory - if you're expecting a large volume of data in your queues, then 1Gb seems quite small.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">How would you like to be able to tune it? There are some parameters of the message store which are configurable, see <a href="http://www.rabbitmq.com/configure.html">http://www.rabbitmq.com/configure.html</a> - but I suspect these will
not do what you want and I'd strongly advise against fiddling with them - as far as we can tell, these are already optimally tuned. In fact, I really suspect that the right thing for you to do is add more memory. Rabbit will use as much memory as possible
to get its job done, so allocating more memory is the way to do IMO.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Tim<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif"">Marek</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Georgia","serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><combo_020.gif>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>