[rabbitmq-discuss] Stuck log rotation in linux?

[Simon MacMullen]

Hi Rob.

The ps snippet you show just shows the tree of processes launched by a 
running RabbitMQ server - nothing out of the ordinary there.

If logrotate is getting stuck, presumably this is due to its invocation 
of /etc/init.d/rabbitmq-server rotate-logs hanging. When it's in the 
stuck state, what does this command return? The logrotate configuration 
file pipes the output of that command to /dev/null - can you redirect it 
somewhere else so we can see what it's doing?

Cheers, Simon

On 26/01/12 21:18, Sutherland, Rob wrote:
> Several times now I’ve run into the issue of rabbitmq getting “stuck”
> during log rotation. Log rotation is done via the usual logrotate daily
> cron using the default logrotate configutaion. This is not simply a case
> of log rotation taking too long as this condition can persist for
> months. This cause log rotation to freeze for the entire system (since
> logrotate correctly refuses to be invoked if there’s already a log
> rotation in progress) and results in log files growing out of control
> (e.g. a 900 MB /var/log/messages file).
>
> Here’s an example of such a process tree (a snippet from ps axjf):
>
> PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
>
> 1 25352 25352 25352 ? -1 Ss 0 0:00 sh -c /usr/sbin/rabbitmq-server >
> /var/log/rabbitmq/startup_log 2> /var/log/rabbitmq/startup_err
>
> 25352 25356 25352 25352 ? -1 S 0 0:00 \_ /bin/sh /usr/sbin/rabbitmq-server
>
> 25356 25363 25352 25352 ? -1 S 0 0:00 \_ su rabbitmq -s /bin/sh -c
> /usr/lib/rabbitmq/bin/rabbitmq-server
>
> 25363 25367 25367 25367 ? -1 Ssl 105 3:56 \_
> /usr/lib64/erlang/erts-5.8.1/bin/beam.smp -W w -K true -A30 -P 1048576
> -- -root /usr/lib64/erlang -prognam
>
> 25367 25479 25479 25479 ? -1 Ss 105 0:00 \_
> /usr/lib64/erlang/lib/os_mon-2.2.5/priv/bin/cpu_sup
>
> 25367 25480 25480 25480 ? -1 Ss 105 0:00 \_ inet_gethost 4
>
> 25480 25481 25480 25480 ? -1 S 105 0:00 \_ inet_gethost 4
>
> Any ideas? I have yet to find a pattern to the appearance of this issue.
>
> Thanks in advance,
>
> Rob
>
> OS: SUSE Linux Enterprise System version 11 SP1
>
> # uname -a
>
> Linux PVHA30node1 2.6.32.46-0.3-default #1 SMP 2011-09-29 17:49:31 +0200
> x86_64 x86_64 x86_64 GNU/Linux
>
> # cat /etc/logrotate.d/ rabbitmq-server
>
> /var/log/rabbitmq/*.log {
>
> weekly
>
> missingok
>
> rotate 20
>
> compress
>
> delaycompress
>
> notifempty
>
> sharedscripts
>
> postrotate
>
> /sbin/service rabbitmq-server rotate-logs > /dev/null
>
> endscript
>
> }
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


-- 
Simon MacMullen
RabbitMQ, VMware