[rabbitmq-discuss] Channel storms

[Matthias Radestock]

Carl,

On 06/09/13 07:50, carlhoerberg wrote:
> We sometimes see node-amqp clients creating thousands and yet thousands of
> channels, bringing a whole cluster to a halt. Normally we detect high
> channel counts by polling the /api/connections endpoint, but when this
> happens, the channels doesn't even belong to a connection, only if you can
> poll /api/channels can you see all the channels (and when the CPU usage is
> 100% and you have 20000 channels this takes hours), but the connection is
> then "unknown"/null.
>
> https://www.dropbox.com/s/e0koajldzxkuqlo/channelstorm.png
>
> When the cluster is overloaded like that it responds to nothing, not even
> rabbitmqctl status or rabbitmqctl delete_user, so a full cluster restart is
> required and then we can block the misbehaving client.
>
> This shows up a lot in the logs when this is happening:
>
> =ERROR REPORT==== 6-Sep-2013::00:25:52 ===
> ** Generic server <0.22899.21> terminating
> ** Last message in was {'$gen_cast',
>                             {method,
>
> {'queue.declare',0,<<>>,false,false,true,true,
>                                     false,[]},
>                                 none,noflow}}
> [...]
> ** Reason for termination ==
> ** {{case_clause,not_found},
>      [{rabbit_channel,handle_method,3},
>       {rabbit_channel,handle_cast,2},
>       {gen_server2,handle_msg,2},
>       {proc_lib,init_p_do_apply,3}]}

Ah, that's a bug, triggered when an exclusive queue is created, and 
during the process of creation the creating connection dies. We will get 
that fixed.

> =WARNING REPORT==== 6-Sep-2013::00:25:52 ===
> Queue {resource,<<"urhyfncz">>,queue,<<"amq.gen-FQZtyp6mpRdmm3-on_LAOQ">>}
> exclusive owner went away

There's a minor bug here too, in that the above should be formatted in a 
more readable way. However, we actually fixed that in 3.1.2, so you 
really shouldn't be seeing that on a server running 3.1.5 (as reported 
in the management screenshot you posted). Was the above log message 
taken from a different server, running an older version?

> RabbitMQ allows this kind of behavior and doesnt report the high
> channel count on connections so we can't automatically detect it
> either :(

On the server, channels can outlive connections for a while. The 
connections are dead; that's why you can't see the channels showing up 
in their counts.

Regards,

Matthias.