>From your suggestions, it looks like I was on the right track. Output is inline.<br><br>Moved to rabbitmq-discuss.<br><br>And again, your guidance is greatly appreciated.<br><br>-Stephen<br><br><div class="gmail_quote">On Thu, Oct 22, 2009 at 1:51 AM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@lshift.net">matthias@lshift.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Stephen,<div class="im"><br>
<br>
Stephen Day wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
After running for few days, limiting queue size on producer side (no memory backpressure), I have noticed that RabbitMQ has restarted several times due to out of memory errors. Currently, I have about 150,000 persistent messages being consumed by 16 clients with qos. Over about 2-6 hours, the RSS and VM size of the erlang process grows unbounded, without the producers adding any messages to the queue (externally limited to 100,000), and the erlang process crashes. I am running version 1.7.0.<br>
</blockquote>
<br></div>
Memory consumption can increase due to gc effects, but not without bounds.<br>
<br>
How many messages does rabbit think there are in the queues - check with 'rabbitmqctl list_queues' - and how big are the messages?<br></blockquote><div><br>$ ~/rabbitmq/sbin/rabbitmqctl list_queues name messages messages_unacknowledged consumers memory<br>
Listing queues ...<br>
<redacted> 0 0 0 2376<br>
<redacted> 27 0 0 27572<br>
<redacted> 0 0 1 2992<br>
<redacted> 149917 16 16 71106232<br><br>The messages are no bigger than 512 bytes, but vary in size.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
How large is the persister log?</blockquote><div><br>$ du -b mnesia/rabbit/*<br>162 mnesia/rabbit/DECISION_TAB.LOG<br>98 mnesia/rabbit/LATEST.LOG<br>8 mnesia/rabbit/rabbit_config.DCD<br>995 mnesia/rabbit/rabbit_durable_exchange.DCD<br>
606 mnesia/rabbit/rabbit_durable_queue.DCD<br>976 mnesia/rabbit/rabbit_durable_queue.DCL<br>1396 mnesia/rabbit/rabbit_durable_route.DCD<br>16720 mnesia/rabbit/rabbit_durable_route.DCL<br>82515919 mnesia/rabbit/rabbit_persister.LOG<br>
82526910 mnesia/rabbit/rabbit_persister.LOG.previous<br>4 mnesia/rabbit/rabbit_serial<br>134 mnesia/rabbit/rabbit_user.DCD<br>192 mnesia/rabbit/rabbit_user_permission.DCD<br>133 mnesia/rabbit/rabbit_vhost.DCD<br>
12383 mnesia/rabbit/schema.DAT<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Now, I was under the impression that this memory problem was due to the number of messages being added, but this may not be the case. What information do you need to troubleshoot this? Are there any erlang tools, similar to valgrind, that can track down this memory usage?<br>
</blockquote>
<br></div>
There are plenty of tools to figure out what's going on in a running Erlang system. You need to know a fair bit about Erlang to use them though.<br>
<br>
As a starting point, remsh into rabbit with "erl -sname sh -remsh rabbit@<host>" (you'll need to run this as the 'rabbitmq' user in a typical rabbit installation) and on the prompt type<br>
memory().<br>
which will tell you at a coarse-grain level where the memory has gone.<br></blockquote><div><br>Here is the memory() output:<br><br>(rabbit@vs-dfw-ctl11)1> memory().<br>[{total,1015558480},<br> {processes,161498128},<br>
{processes_used,161494240},<br> {system,854060352},<br> {atom,516217},<br> {atom_used,492146},<br> {binary,778446312},<br> {code,3860276},<br> {ets,70066180}]<br><br>Right now, the system is using around 1GB of memory, mostly concentrated in binary allocation. I was looking at this yesterday as well, and the process memory hasn't changed much.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
Regards,<br><font color="#888888">
<br>
Matthias.<br></font>
PS: could we move this discussion onto rabbitmq-discuss, please?<br>
</blockquote></div><br>