<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-family:"Courier New"'>Several times now I’ve run into the issue of rabbitmq getting “stuck” during log rotation. Log rotation is done via the usual logrotate daily cron using the default logrotate configutaion. This is not simply a case of log rotation taking too long as this condition can persist for months. This cause log rotation to freeze for the entire system (since logrotate correctly refuses to be invoked if there’s already a log rotation in progress) and results in log files growing out of control (e.g. a 900 MB /var/log/messages file).<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Here’s an example of such a process tree (a snippet from ps axjf):<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1 25352 25352 25352 ? -1 Ss 0 0:00 sh -c /usr/sbin/rabbitmq-server > /var/log/rabbitmq/startup_log 2> /var/log/rabbitmq/startup_err<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>25352 25356 25352 25352 ? -1 S 0 0:00 \_ /bin/sh /usr/sbin/rabbitmq-server<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>25356 25363 25352 25352 ? -1 S 0 0:00 \_ su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmq-server <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>25363 25367 25367 25367 ? -1 Ssl 105 3:56 \_ /usr/lib64/erlang/erts-5.8.1/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -prognam<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>25367 25479 25479 25479 ? -1 Ss 105 0:00 \_ /usr/lib64/erlang/lib/os_mon-2.2.5/priv/bin/cpu_sup<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>25367 25480 25480 25480 ? -1 Ss 105 0:00 \_ inet_gethost 4<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>25480 25481 25480 25480 ? -1 S 105 0:00 \_ inet_gethost 4<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Any ideas? I have yet to find a pattern to the appearance of this issue.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Thanks in advance,<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Rob<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>OS: SUSE Linux Enterprise System version 11 SP1<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'># uname -a<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Linux PVHA30node1 2.6.32.46-0.3-default #1 SMP 2011-09-29 17:49:31 +0200 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'># cat /etc/logrotate.d/</span> <span style='font-family:"Courier New"'>rabbitmq-server<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>/var/log/rabbitmq/*.log {<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> weekly<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> missingok<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> rotate 20<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> compress<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> delaycompress<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> notifempty<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> sharedscripts<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> postrotate<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> /sbin/service rabbitmq-server rotate-logs > /dev/null<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> endscript<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>}<o:p></o:p></span></p></div></body></html>