[rabbitmq-discuss] memory consumption during message delivery

Wed Apr 3 13:35:56 BST 2013

Marek,

On 3 Apr 2013, at 13:16, Cermak, Marek wrote:

> Hi everybody,
> I did some measurements of memory consumption of RMQ (v3.0.2) during sending and delivering messages and found an interesting behaviour during delivery. In short: messages were stored in mnesia transient storage on disk and when a consumer starts to consume them all of them were read to memory in one chunk which was almost deadly for RMQ and the server.

The mnesia database is used to store schema elements (queue definitions, exchanges, users, etc), however messages are *never* stored in mnesia, they're stored in a custom file system storage layer - the persister -  which was purpose written for RabbitMQ.

>  
> The scenario was 1+2+8+10 pairs of bound direct exchanges and queues each received messages of particular size (1kB, 64 kB, 1 MB and 8MB) and after a while consumer was started to 'empty' each of the queues. The consumer of the queue with biggest messages triggered the avalanche of data being transferred from disk to memory. Similar behaviour can be seen always when you fill the queue up to the roof with bigger messages and start the consumer.
[snip]
> Tested with 1G of memory, 64b Debian GNU/linux, Erlang R15B, RMQ v3.0.2.
>  
> Is it an expected behaviour? Can it be somehow tuned/adjusted? Thanks.
>  

Well if you've got a bunch of data on disk that needs to be delivered to a consumer later on, it'll need to be read into memory in order to be transferred onto the wire. One obvious way to avoid getting into this position would be to allocate a good deal more memory - if you're expecting a large volume of data in your queues, then 1Gb seems quite small.

How would you like to be able to tune it? There are some parameters of the message store which are configurable, see http://www.rabbitmq.com/configure.html - but I suspect these will not do what you want and I'd strongly advise against fiddling with them - as far as we can tell, these are already optimally tuned. In fact, I really suspect that the right thing for you to do is add more memory. Rabbit will use as much memory as possible to get its job done, so allocating more memory is the way to do IMO.

Cheers,
Tim

> Marek
>  
>  
> <combo_020.gif>_______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130403/546ac7c9/attachment.htm>