[rabbitmq-discuss] rabbitmq memory usage

[Ceri Storey]

(06/06/13 14:45), Sean Allen wrote:
> I'm trying to understand memory usage in rabbitmq. Pointers to
> relevant docs, salient comments etc greatly welcomed. We have a number
> of batch jobs that are handled by one process putting information on a
> queue and then storm spouts read the info off and process. In the case
> of one job, we put about 5.9 million integers on a queue as individual
> messages and process. When those 5.9 go on, rabbitmq usage jumps up by
> about a gig.
>
> Currently with 5.6 million still in the queue, we have the following
> memory usage on the node as determind by rabbitmqctl status:
>
>  {memory,
>      [{total,1178020816},
>       {processes,131791280},
>       {processes_used,130916744},
>       {system,1046229536},
>       {atom,1338193},
>       {atom_used,1314590},
>       {binary,12936792},
>       {code,15696033},
>       {ets,975434504}]},
>
> Can anyone shed light on the high level of memory usage and what we
> could do to cut down on it?

I'm sure that one of the Rabbit team themselves can shed a bit more
light on this, but AIUI RabbitMQ uses ETS in-memory tables to index the
queues, so with some back of the envelope arithmetic your rabbit is
using approximately 175bytes per message. So on the face of it, that
might look a bit odd, but doesn't seem unreasonable.

Are you seeing any concrete issues because of this, or are you more
concerned about resource usage? Also, do you anticipate having to
process batches many factors larger than that?