[rabbitmq-discuss] RabbitMQ crash caused by channel leak?

[Matthew Sackman]

On Thu, Mar 11, 2010 at 02:28:48PM -0600, Ian Ragsdale wrote:
> =CRASH REPORT==== 11-Mar-2010::08:13:21 ===
>   crasher:
>     pid: <0.16638.1>
>     registered_name: []
>     exception error: a system limit has been reached
>       in function  spawn/3

Yes, I think you've probably run out of processes. The default process
limit in erlang is 32768, but this can be raised up to 134217727. To set
this for Rabbit, in your rabbitmq.conf file (which under Linux will be
at /etc/rabbitmq/rabbitmq.conf) add:

SERVER_START_ARGS="+P 1000000"

Which will get you 1000,000 processes. Note that the additional
accounting done by the erlang VM will mean that you'll see more memory
used.

> Based on the number of channels in the logged dictionary, I'm guessing I hit a limit on the number of channels, which I'm guessing was the cause of the crash.  Does this sound like a likely cause?  I've identified and removed the code that was creating all the channels, but I'm concerned that it appears to be so easy for a single rogue client to take down the entire server.  Is there a way for me to prevent this?

Mmmm, that's a good point, and no, there's no such knob. In
rabbit_reader.erl, at around line 589, you should find the following
code:

    ok = send_on_channel0(
           Sock,
           #'connection.tune'{channel_max = 0,
                              %% set to zero once QPid fix their negotiation
                              frame_max = 131072,
                              heartbeat = 0}),

That channel_max = 0 implies no limit on the number of channels (well,
it's a 16-bit uint, so 65536). If you set that to a lower number then
well-behaved clients may take note of that. However it's still not
enforced at the server. For that, you'll need to either add to the last
head of handle_frame in rabbit_reader (at the bottom, before it calls
send_to_new_channel). The difficulty here is that channels don't need to
be allocated in the correct order and can be reused. Gaps can appear etc
etc, so it's a bit harder than just looking at the channels number, you
actually have to count. I'll raise a bug for this and we'll fix it in
due course.

Matthew