[rabbitmq-discuss] rabbitmq memory leak in 2.7.0/2.8.1

Thu May 10 05:09:56 BST 2012

We are in the advanced stage of performance testing of a demanding
application.  This application consists of a large number (2357)
processes that connect to a local RabbitMQ server.

RabbitMQ version: 2.7.0-1
Erlang version: Erlang R14B02 (erts-5.8.3) [source] [64-bit] [smp:
24:24] [rq:24] [async-threads:0] [hipe] [kernel-poll:false]
Linux kernel: Linux plt68w98m7 2.6.18-212.el5 #1 SMP Wed Aug 11
23:22:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Hardware: Dell R610 with 24GB memory and 24 CPUs.
RabbitMQ server configuration:  default configuration from the rpm
package, no customization.

2357 application processes are using 40 distinct durable queues, with
each process listening to a unique queue. Below is the distribution of
number of processes listening for each queue:
numprocs=1
numprocs=1
numprocs=1
numprocs=1
numprocs=300
numprocs=300
numprocs=10
numprocs=40
numprocs=10
numprocs=300
numprocs=3
numprocs=10
numprocs=10
numprocs=300
numprocs=10
numprocs=20
numprocs=60
numprocs=60
numprocs=4
numprocs=5
numprocs=300
numprocs=30
numprocs=10
numprocs=1
numprocs=60
numprocs=160
numprocs=15
numprocs=80
numprocs=10
numprocs=10
numprocs=160
numprocs=5
numprocs=10
numprocs=10
numprocs=10
numprocs=5
numprocs=5
numprocs=15
numprocs=10
numprocs=5

If an application process needs to send messages to another
application, the sender process creates a unique "auto-delete
exclusive" reply queue. The queue is kept until the process is
shutdown.

When the application processes are processing large number of
messages, you would see RabbitMQ "beam.smp" consume increasingly more
memory.
When all application processes are terminated and no messages remain
in any queues (viewed via RabbitMQ management http interface), the
memory consumption of beam.smp would not go down.

When the application processes are started and another performance run
is performed, the memory usage of beam.smp keeps going higher.

further we ran the same tests with:
erlang-R15B01-0.x86_64.rpm
rabbitmq-server-2.8.1-0.i386.rpm

I have a tester that allows one service call to fan out 120 service
calls.
When I ran the tester with a loop count of 10,000, it would generate
about 1.2M calls. After the test is finished, you end up with 475MB
virtual memory and 335MB resident memory. In the middle of the
testing, I have seen memory usage up to 700M virtual memory and 400MB
resident memory.

If you restart the test, the memory would keep going up. Imaging that
we want to keep our servers up for a release cycle, we would run  out
of memory.

Any pointers/suggestions will be very helpful.
regards,
-monish