[rabbitmq-discuss] RabbmitMQ consuming CPU and Memory

[Matthias Radestock]

Paul,

On 10/05/12 16:40, Paul Gross wrote:
> We are seeing weird issues with RabbmitMQ and CPU/Memory. Basically, we
> run tests locally and on our build server. In both cases, we create a
> couple of queues, push messages through (not many, maybe a few hundred),
> and consume them. This works fine for a few days.
>
> Then, all of a sudden, RabbitMQ goes nuts. It consumes all available CPU
> and memory until it crashes the server. As far as we can tell, nothing
> is different right before this happens. We have to manually kill
> RabbitMQ, and when it comes back up, the queue has less than 10
> messages. The last time this happened, the only thing in the log was
> "Mnesia is overloaded."

The "mnesia is overloaded" message could be an indication of a high rate 
of queue/exchange/binding creation/deletion. But I suspect it may also 
be triggered at low creation/deletion rates when rabbit is generally 
busy, so this may be a red herring.

> This happens both locally on a MacPro (10.7.3) and on our build server
> (Ubuntu 10.04).
>
> These images are from munin, and give an idea of what's going on:
>
> http://dl.dropbox.com/u/15440955/rabbitmq-cpu.png
> http://dl.dropbox.com/u/15440955/rabbitmq-memory.png
>
> We are looking for help on how to debug this problem. We cannot directly
> reproduce the problem, but it happens every couple of days.
>
> Here is our setup on the build server:
>     {rabbit,"RabbitMQ","2.7.1"},
>     {erlang_version,"Erlang R13B03 ..."}
>     [...]
> And here is our setup on the MacPro:
>     {rabbit,"RabbitMQ","2.7.1"},
>     {erlang_version,"Erlang R15B ..."}

There have been numerous stability improvements since 2.7.1, so I 
suggest you upgrade to the latest rabbit version (2.8.2). Also, R13B03 
is quite ancient, so you should really upgrade that too - R15B01 is the 
latest.

If you still see the same behaviour then try to set up some alarms that 
catch the problem in the early stages and grab the output of 
'rabbitmqctl report'.


Regards,

Matthias.