<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body>
<pre style="word-wrap:break-word; font-size:10.0pt; font-family:Tahoma; color:black">Hello
You mentioned a cluster.
Is this happening on every box ?
Also, is the rabbit process running as "foobar" ?
Jeff
Matt Pietrek <mpietrek@skytap.com> wrote:
</pre>
<div>
<div dir="ltr">Here you go Mark. The file date/times line up with when I last saw logging in the files:<br>
<br>
23:14 PROD mpietrek@foomq1:/proc/16050$ sudo ls -latr fd<br>
total 0<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 2 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbitmq-server.log">foomq1.foo.bar.com/rabbitmq-server.log</a><br>
dr-xr-xr-x 7 foobar foobar 0 2013-06-02 11:50 ..<br>
dr-x------ 2 foobar foobar 0 2013-06-02 11:50 .<br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 9 -> pipe:[122339989]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 8 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbit@foomq1sasl.log">foomq1.foo.bar.com/rabbit@foomq1sasl.log</a><br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 7 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbit@foomq1.log">foomq1.foo.bar.com/rabbit@foomq1.log</a><br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 6 -> socket:[122339982]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 5 -> socket:[122339980]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 4 -> pipe:[122339970]<br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 3 -> pipe:[122339970]<br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 17 -> socket:[122340310]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 16 -> socket:[122340285]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 15 -> /foobar/var/lib/rabbit@foomq1/msg_store_persistent/397.rdq<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 14 -> /foobar/var/lib/rabbit@foomq1/msg_store_transient/0.rdq<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 13 -> /foobar/var/lib/rabbit@foomq1/queues/D1KLLKOJLVJ4YZGYLP2D32A9W/journal.jif<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 12 -> pipe:[122339990]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 11 -> /foobar/var/lib/rabbit@foomq1/LATEST.LOG<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 10 -> socket:[122339997]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 1 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbitmq-server.log">foomq1.foo.bar.com/rabbitmq-server.log</a><br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 0 -> /dev/null<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:51 18 -> socket:[122340314]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 12:28 21 -> /foobar/var/lib/rabbit@foomq1/queues/ETWUA9HXDWBIXFTEF0DVH5FBZ/journal.jif<br>
lr-x------ 1 foobar foobar 64 2013-06-02 12:28 19 -> /foobar/var/lib/rabbit@foomq1/queues/22TABTKBIJ6DQCA5GC6EZ0B8L/journal.jif<br>
l-wx------ 1 foobar foobar 64 2013-06-02 13:12 22 -> /foobar/var/lib/rabbit@foomq1/queues/A9AWP930R556ERL81W9PYEELL/journal.jif<br>
lrwx------ 1 foobar foobar 64 2013-06-02 13:12 20 -> /foobar/var/lib/rabbit@foomq1/queues/A8EPKFQ4VCO5A3APTTWYEQRFT/journal.jif<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Jul 16, 2013 at 12:19 PM, Jeffery, Mark <span dir="ltr">
<<a href="mailto:MJeffery@fnb.co.za" target="_blank">MJeffery@fnb.co.za</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div>
<pre style="font-size:10.0pt; font-family:Tahoma; word-wrap:break-word">Hello
Assuming Linux, what does ls -latr show in the fd directory in the rabbitmq process's directory under /proc ?
Matt Pietrek <<a href="mailto:mpietrek@skytap.com" target="_blank">mpietrek@skytap.com</a>> wrote:
</pre>
<div>
<div class="h5">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>This is admittedly a shot in the dark, and with limited repro details, but asking if somebody else has seen this.<br>
</div>
<br>
</div>
We run our clustered RabbitMQ 3.0.2 brokers in production without restarts for months at a time. Everything is looking good - No memory leaks, no out-of-disk-space, etc...<br>
<br>
</div>
However, in looking at the rabbit@<node>.log and rabbit@<node>-sasl.log files, I noticed that at some point (e.g. a month earlier), no new entries were being added to the log file. Normally there's a steady stream of new connections and connection drops.<br>
<br>
I've listed the open file handles from the rabbitmq beam instance and I still see live file handles for the log files.<br>
<br>
</div>
Anybody have any idea what might be happening? And better still, how we might kick things without restarting the broker?<br>
<br>
</div>
Thanks,<br>
<br>
Matt<br>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<font face="Times New Roman" size="3"><span lang="EN-US" style="font-size:12pt">
<p class="MsoNormal" align="left" style="margin:0cm 0cm 0pt 36pt"><font face="Arial"><span style="font-size:10pt; font-family:Arial">To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
<br>
<a href="https://www.fnb.co.za/disclaimer.html" target="_blank">https://www.fnb.co.za/disclaimer.html</a>
<br>
<br>
</span></font></p>
<u></u><u></u></span></font><font face="Arial"><span style="font-size:10pt; font-family:Arial">
<p class="MsoNormal" style="margin:0cm 0cm 0pt 36pt"><font face="Arial"><span style="font-size:10pt; font-family:Arial">If you are unable to access the Disclaimer, send a blank e-mail to
</span></font><font face="Arial"><span style="font-size:10pt; font-family:Arial"><a href="mailto:firstrandbankdisclaimer@fnb.co.za" target="_blank">firstrandbankdisclaimer@fnb.co.za</a> and we will send you a copy of the Disclaimer.<u></u><u></u></span></font></p>
<p class="MsoNormal" align="left" style="margin:0cm 0cm 0pt 36pt"></p>
</span></font></div>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<FONT face="Times New Roman" size=3><SPAN lang=EN-US
style="FONT-SIZE: 12pt; mso-ansi-language: EN-US">
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt 36pt" align=left><FONT face=Arial
size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">To read FirstRand
Bank's Disclaimer for this email click on the following address or copy into
your Internet browser: <BR><A
href="https://www.fnb.co.za/disclaimer.html">https://www.fnb.co.za/disclaimer.html</A>
<BR style="mso-special-character: line-break"><BR
style="mso-special-character: line-break"></P><O:P></O:P></SPAN></FONT><FONT
face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt 36pt"><FONT face=Arial
size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">If you are unable to
access the Disclaimer, send a blank e-mail to </SPAN></FONT><FONT face=Arial
size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><A
href="mailto:firstrandbankdisclaimer@fnb.co.za">firstrandbankdisclaimer@fnb.co.za</A>
and we will send you a copy of the Disclaimer.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt 36pt"
align=left></SPAN></FONT></SPAN></FONT></P>
</body>
</html>