<html><head><base href="x-msg://591/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Marek,<div><br><div><div>On 3 Apr 2013, at 13:16, Cermak, Marek wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; ">Hi everybody,<o:p></o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; ">I did some measurements of memory consumption of RMQ (v3.0.2) during sending and delivering messages and found an interesting behaviour during delivery. In short: messages were stored in mnesia transient storage on disk and when a consumer starts to consume them all of them were read to memory in one chunk which was almost deadly for RMQ and the server.<o:p></o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; "></span></div></div></div></span></blockquote><div><br></div><div>The mnesia database is used to store schema elements (queue definitions, exchanges, users, etc), however messages are *never* stored in mnesia, they're stored in a custom file system storage layer - the persister - which was purpose written for RabbitMQ.</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; "><o:p> </o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; ">The scenario was 1+2+8+10 pairs of bound direct exchanges and queues each received messages of particular size (1kB, 64 kB, 1 MB and 8MB) and after a while consumer was started to 'empty' each of the queues. The consumer of the queue with biggest messages triggered the avalanche of data being transferred from disk to memory. Similar behaviour can be seen always when you fill the queue up to the roof with bigger messages and start the consumer.</span></div></div></div></span></blockquote>[snip]<br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; ">Tested with 1G of memory, 64b Debian GNU/linux, Erlang R15B, RMQ v3.0.2.<o:p></o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; "><o:p> </o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; ">Is it an expected behaviour? Can it be somehow tuned/adjusted? Thanks.<o:p></o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; "><o:p> </o:p></span></div></div></div></span></blockquote><div><br></div><div>Well if you've got a bunch of data on disk that needs to be delivered to a consumer later on, it'll need to be read into memory in order to be transferred onto the wire. One obvious way to avoid getting into this position would be to allocate a good deal more memory - if you're expecting a large volume of data in your queues, then 1Gb seems quite small.</div><div><br></div><div>How would you like to be able to tune it? There are some parameters of the message store which are configurable, see <a href="http://www.rabbitmq.com/configure.html">http://www.rabbitmq.com/configure.html</a> - but I suspect these will not do what you want and I'd strongly advise against fiddling with them - as far as we can tell, these are already optimally tuned. In fact, I really suspect that the right thing for you to do is add more memory. Rabbit will use as much memory as possible to get its job done, so allocating more memory is the way to do IMO.</div><div><br></div><div>Cheers,</div><div>Tim</div><br><blockquote type="cite"><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; ">Marek<o:p></o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; "><o:p> </o:p></span></div><div style="margin-top: 0mm; margin-right: 0mm; margin-left: 0mm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><span style="font-size: 10pt; font-family: Georgia, serif; "><o:p> </o:p></span></div></div><span><combo_020.gif></span>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" style="color: blue; text-decoration: underline; ">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" style="color: blue; text-decoration: underline; ">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br></div></blockquote></div><br></div></body></html>