[rabbitmq-discuss] rabbitmq memory usage

Thu Jun 6 16:37:33 BST 2013

On Thu, Jun 6, 2013 at 10:03 AM, Ceri Storey <ceri at lshift.net> wrote:

> (06/06/13 14:45), Sean Allen wrote:
> > I'm trying to understand memory usage in rabbitmq. Pointers to
> > relevant docs, salient comments etc greatly welcomed. We have a number
> > of batch jobs that are handled by one process putting information on a
> > queue and then storm spouts read the info off and process. In the case
> > of one job, we put about 5.9 million integers on a queue as individual
> > messages and process. When those 5.9 go on, rabbitmq usage jumps up by
> > about a gig.
> >
> > Currently with 5.6 million still in the queue, we have the following
> > memory usage on the node as determind by rabbitmqctl status:
> >
> >  {memory,
> >      [{total,1178020816},
> >       {processes,131791280},
> >       {processes_used,130916744},
> >       {system,1046229536},
> >       {atom,1338193},
> >       {atom_used,1314590},
> >       {binary,12936792},
> >       {code,15696033},
> >       {ets,975434504}]},
> >
> > Can anyone shed light on the high level of memory usage and what we
> > could do to cut down on it?
>
> I'm sure that one of the Rabbit team themselves can shed a bit more
> light on this, but AIUI RabbitMQ uses ETS in-memory tables to index the
> queues, so with some back of the envelope arithmetic your rabbit is
> using approximately 175bytes per message. So on the face of it, that
> might look a bit odd, but doesn't seem unreasonable.
>
> Are you seeing any concrete issues because of this, or are you more
> concerned about resource usage? Also, do you anticipate having to
> process batches many factors larger than that?
>

We are adding more batch processes and with 2 gigs per node, we are able to
push past that.
So yes, that are some concerns there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130606/15cd8527/attachment.htm>