<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">(06/06/13 16:37), Sean Allen wrote:<br>
</div>
<blockquote
cite="mid:CABm-WmfTWSOj=d5+6_i7zsqW1D223MZF_Lc_AwJxDxKQ0ObOUQ@mail.gmail.com"
type="cite">
<div dir="ltr">On Thu, Jun 6, 2013 at 10:03 AM, Ceri Storey <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:ceri@lshift.net" target="_blank">ceri@lshift.net</a>></span>
wrote:<br>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb">
<div class="h5">(06/06/13 14:45), Sean Allen wrote:<br>
> I'm trying to understand memory usage in
rabbitmq. Pointers to<br>
> relevant docs, salient comments etc greatly
welcomed. We have a number<br>
> of batch jobs that are handled by one process
putting information on a<br>
> queue and then storm spouts read the info off and
process. In the case<br>
> of one job, we put about 5.9 million integers on
a queue as individual<br>
> messages and process. When those 5.9 go on,
rabbitmq usage jumps up by<br>
> about a gig.<br>
><br>
> ....<br>
</div>
</div>
I'm sure that one of the Rabbit team themselves can shed a
bit more<br>
light on this, but AIUI RabbitMQ uses ETS in-memory tables
to index the<br>
queues, so with some back of the envelope arithmetic your
rabbit is<br>
using approximately 175bytes per message. So on the face
of it, that<br>
might look a bit odd, but doesn't seem unreasonable.<br>
<br>
Are you seeing any concrete issues because of this, or are
you more<br>
concerned about resource usage? Also, do you anticipate
having to<br>
process batches many factors larger than that?<br>
</blockquote>
<div><br>
</div>
<div style="">We are adding more batch processes and with 2
gigs per node, we are able to push past that.</div>
<div style="">So yes, that are some concerns there. <br>
</div>
</div>
</div>
</div>
</blockquote>
Okay--so the biggest risk that I can foresee for rabbit itself
Rabbit will start hitting it's high watermark for memory usage, and
start exerting back-pressure by not reading from the socket. Once
the storm AMQP spout has drained enough messages from the queue,
Rabbit's memory usage will drop, and start reading from publishers
once again. <br>
<br>
So, the question in my mind, would be how well the publishing
applications deals with the back pressure; if it will just wait
blocking on the socket, you should be fine. Assuming that the spout
continuously consumes messages, I don't <i>believe</i> there would
be a problem; but it'll be worth checking that the heartbeat period
is longer than it takes for rabbit to unblock connections and
process the buffered data. (Rabbit logs to <tt>rabbit.log</tt> when
it hits and clears the high water mark alarms).<br>
<br>
I'd be tempted to stress test the whole setup, and see how far you
can push it before it breaks bearing in mind the above caveats.<br>
<br>
Hopefully that's of some use, at least.<br>
</body>
</html>