[rabbitmq-discuss] Many RabbitMQ Status Processes

[Tim Watson]

Hi Eric,

On 19 Mar 2013, at 16:18, Eric Berg wrote:
> Hi! I have a 2 node cluster running for a prototype we are developing, it handles over 1000 queues and over 1000 msg/s when running at full speed. We had an issue recently with some run away processes that published messages from production app servers vi the php library:
[snip]
> The cause of that is still under investigation, but it left many thousands of 'stale' idle connections that rabbit was maintaining.

How did you identify that - using the management UI? What version of rabbit are you running and on which os/platform?

> Attempting to stop/start each node hangs indefinitely, the only way get a full restart of the server is to actually restart the hardware. I see 58 'status' processes attributed to rabbit:
> 
> rabbitmq 16066 16033  0 Mar18 ?        00:00:00 inet_gethost 4
> rabbitmq 16067 16066  0 Mar18 ?        00:00:00 inet_gethost 4
> root     17713 20340  0 05:27 ?        00:00:00 /bin/sh /etc/init.d/rabbitmq-server status
> root     17716 17713  0 05:27 ?        00:00:00 /bin/sh /usr/sbin/rabbitmqctl status
> root     17725 17716  0 05:27 ?        00:00:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl  "status"
> rabbitmq 17726 17725  0 05:27 ?        00:00:00 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.0.1/sbin/../ebin -noshell -noinput -hidden -sname rabbitmqctl17726 -s rabbit_control_main -nodename rabbit at rabbit -extra status
> 

Where's the 58 coming from?

> 
> When stopping the node I see this in the log file:
> =INFO REPORT==== 19-Mar-2013::11:18:32 ===
> Stopping RabbitMQ
> 
> =INFO REPORT==== 19-Mar-2013::11:18:32 ===
>     application: rabbitmq_management_visualiser
>     exited: stopped
>     type: temporary
> 
> =INFO REPORT==== 19-Mar-2013::11:18:32 ===
>     application: rabbitmq_federation_management
>     exited: stopped
>     type: temporary
> 

Those look perfectly normal. I thought you said you couldn't stop the node(s) at all?

> 
> Has anyone see issues with many thousands of stale connections, in combination with many status processes and hanging stop/start? Currently there are are no producers or consumers connected to this box.

The 'status' processes are only ever launched by the init daemon or by a user issuing `rabbitmq-server status` - are you sure you don't have some monitoring infrastructure that is triggering these commands?

Cheers,
Tim

> 
> Thanks!
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss