[rabbitmq-discuss] rabbitmq memory usage

[Ceri Storey]

(06/06/13 16:37), Sean Allen wrote:
> On Thu, Jun 6, 2013 at 10:03 AM, Ceri Storey <ceri at lshift.net
> <mailto:ceri at lshift.net>> wrote:
>
>     (06/06/13 14:45), Sean Allen wrote:
>     > I'm trying to understand memory usage in rabbitmq. Pointers to
>     > relevant docs, salient comments etc greatly welcomed. We have a
>     number
>     > of batch jobs that are handled by one process putting
>     information on a
>     > queue and then storm spouts read the info off and process. In
>     the case
>     > of one job, we put about 5.9 million integers on a queue as
>     individual
>     > messages and process. When those 5.9 go on, rabbitmq usage jumps
>     up by
>     > about a gig.
>     >
>     > ....
>     I'm sure that one of the Rabbit team themselves can shed a bit more
>     light on this, but AIUI RabbitMQ uses ETS in-memory tables to
>     index the
>     queues, so with some back of the envelope arithmetic your rabbit is
>     using approximately 175bytes per message. So on the face of it, that
>     might look a bit odd, but doesn't seem unreasonable.
>
>     Are you seeing any concrete issues because of this, or are you more
>     concerned about resource usage? Also, do you anticipate having to
>     process batches many factors larger than that?
>
>
> We are adding more batch processes and with 2 gigs per node, we are
> able to push past that.
> So yes, that are some concerns there.
Okay--so the biggest risk that I can foresee for rabbit itself Rabbit
will start hitting it's high watermark for memory usage, and start
exerting back-pressure by not reading from the socket. Once the storm
AMQP spout has drained enough messages from the queue, Rabbit's memory
usage will drop, and start reading from publishers once again.

So, the question in my mind, would be how well the publishing
applications deals with the back pressure; if it will just wait blocking
on the socket, you should be fine. Assuming that the spout continuously
consumes messages, I don't /believe/ there would be a problem; but it'll
be worth checking that the heartbeat period is longer than it takes for
rabbit to unblock connections and process the buffered data. (Rabbit
logs to rabbit.log when it hits and clears the high water mark alarms).

I'd be tempted to stress test the whole setup, and see how far you can
push it before it breaks bearing in mind the above caveats.

Hopefully that's of some use, at least.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130606/26a019b4/attachment.htm>