<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 6, 2013 at 10:03 AM, Ceri Storey <span dir="ltr"><<a href="mailto:ceri@lshift.net" target="_blank">ceri@lshift.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">(06/06/13 14:45), Sean Allen wrote:<br>
> I'm trying to understand memory usage in rabbitmq. Pointers to<br>
> relevant docs, salient comments etc greatly welcomed. We have a number<br>
> of batch jobs that are handled by one process putting information on a<br>
> queue and then storm spouts read the info off and process. In the case<br>
> of one job, we put about 5.9 million integers on a queue as individual<br>
> messages and process. When those 5.9 go on, rabbitmq usage jumps up by<br>
> about a gig.<br>
><br>
> Currently with 5.6 million still in the queue, we have the following<br>
> memory usage on the node as determind by rabbitmqctl status:<br>
><br>
> {memory,<br>
> [{total,1178020816},<br>
> {processes,131791280},<br>
> {processes_used,130916744},<br>
> {system,1046229536},<br>
> {atom,1338193},<br>
> {atom_used,1314590},<br>
> {binary,12936792},<br>
> {code,15696033},<br>
> {ets,975434504}]},<br>
><br>
> Can anyone shed light on the high level of memory usage and what we<br>
> could do to cut down on it?<br>
<br>
</div></div>I'm sure that one of the Rabbit team themselves can shed a bit more<br>
light on this, but AIUI RabbitMQ uses ETS in-memory tables to index the<br>
queues, so with some back of the envelope arithmetic your rabbit is<br>
using approximately 175bytes per message. So on the face of it, that<br>
might look a bit odd, but doesn't seem unreasonable.<br>
<br>
Are you seeing any concrete issues because of this, or are you more<br>
concerned about resource usage? Also, do you anticipate having to<br>
process batches many factors larger than that?<br></blockquote><div><br></div><div style>We are adding more batch processes and with 2 gigs per node, we are able to push past that.</div><div style>So yes, that are some concerns there. </div>
<div style><br></div></div>
</div></div>