<div dir="ltr"><div><div><div><div><br>Jeff,<br><br>The "foobar" in the listing I provided is the result of my search/replace obfuscation. Don't want internal server names, account names, etc. revealed unnecessarily.<br>
<br></div>The logging stopped on only one node of the cluster. The other node continued to log as expected.<br><br></div>Interestingly, on the broker that still logged, I see this message at the time of the last log entry of the non-logging machine:<br>
<br>=INFO REPORT==== 2-Jun-2013::11:50:57 ===<br>rabbit on node rabbit@foobar up<br><br></div>(again, where foobar is obfuscated).<br><br></div>Digging around some other logs at the time, I see there was a mnesia/network split issue just proceeding this. However, the broker now looks to be happily a part of the cluster, participating in mirrored queues, and with a reported uptime matching that of the "INFO REPORT" above.<br>
<div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 16, 2013 at 10:51 PM, Jeffery, Mark <span dir="ltr"><<a href="mailto:MJeffery@fnb.co.za" target="_blank">MJeffery@fnb.co.za</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<pre style="font-size:10.0pt;font-family:Tahoma;word-wrap:break-word">Hello
You mentioned a cluster.
Is this happening on every box ?
Also, is the rabbit process running as "foobar" ?
Jeff
Matt Pietrek <<a href="mailto:mpietrek@skytap.com" target="_blank">mpietrek@skytap.com</a>> wrote:
</pre><div><div class="h5">
<div>
<div dir="ltr">Here you go Mark. The file date/times line up with when I last saw logging in the files:<br>
<br>
23:14 PROD mpietrek@foomq1:/proc/16050$ sudo ls -latr fd<br>
total 0<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 2 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbitmq-server.log" target="_blank">foomq1.foo.bar.com/rabbitmq-server.log</a><br>
dr-xr-xr-x 7 foobar foobar 0 2013-06-02 11:50 ..<br>
dr-x------ 2 foobar foobar 0 2013-06-02 11:50 .<br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 9 -> pipe:[122339989]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 8 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbit@foomq1sasl.log" target="_blank">foomq1.foo.bar.com/rabbit@foomq1sasl.log</a><br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 7 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbit@foomq1.log" target="_blank">foomq1.foo.bar.com/rabbit@foomq1.log</a><br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 6 -> socket:[122339982]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 5 -> socket:[122339980]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 4 -> pipe:[122339970]<br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 3 -> pipe:[122339970]<br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 17 -> socket:[122340310]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 16 -> socket:[122340285]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 15 -> /foobar/var/lib/rabbit@foomq1/msg_store_persistent/397.rdq<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 14 -> /foobar/var/lib/rabbit@foomq1/msg_store_transient/0.rdq<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 13 -> /foobar/var/lib/rabbit@foomq1/queues/D1KLLKOJLVJ4YZGYLP2D32A9W/journal.jif<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 12 -> pipe:[122339990]<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 11 -> /foobar/var/lib/rabbit@foomq1/LATEST.LOG<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 10 -> socket:[122339997]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 1 -> /foobar/logs/<a href="http://foomq1.foo.bar.com/rabbitmq-server.log" target="_blank">foomq1.foo.bar.com/rabbitmq-server.log</a><br>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 0 -> /dev/null<br>
lrwx------ 1 foobar foobar 64 2013-06-02 11:51 18 -> socket:[122340314]<br>
l-wx------ 1 foobar foobar 64 2013-06-02 12:28 21 -> /foobar/var/lib/rabbit@foomq1/queues/ETWUA9HXDWBIXFTEF0DVH5FBZ/journal.jif<br>
lr-x------ 1 foobar foobar 64 2013-06-02 12:28 19 -> /foobar/var/lib/rabbit@foomq1/queues/22TABTKBIJ6DQCA5GC6EZ0B8L/journal.jif<br>
l-wx------ 1 foobar foobar 64 2013-06-02 13:12 22 -> /foobar/var/lib/rabbit@foomq1/queues/A9AWP930R556ERL81W9PYEELL/journal.jif<br>
lrwx------ 1 foobar foobar 64 2013-06-02 13:12 20 -> /foobar/var/lib/rabbit@foomq1/queues/A8EPKFQ4VCO5A3APTTWYEQRFT/journal.jif<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Jul 16, 2013 at 12:19 PM, Jeffery, Mark <span dir="ltr">
<<a href="mailto:MJeffery@fnb.co.za" target="_blank">MJeffery@fnb.co.za</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<pre style="font-size:10.0pt;font-family:Tahoma;word-wrap:break-word">Hello
Assuming Linux, what does ls -latr show in the fd directory in the rabbitmq process's directory under /proc ?
Matt Pietrek <<a href="mailto:mpietrek@skytap.com" target="_blank">mpietrek@skytap.com</a>> wrote:
</pre>
<div>
<div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>This is admittedly a shot in the dark, and with limited repro details, but asking if somebody else has seen this.<br>
</div>
<br>
</div>
We run our clustered RabbitMQ 3.0.2 brokers in production without restarts for months at a time. Everything is looking good - No memory leaks, no out-of-disk-space, etc...<br>
<br>
</div>
However, in looking at the rabbit@<node>.log and rabbit@<node>-sasl.log files, I noticed that at some point (e.g. a month earlier), no new entries were being added to the log file. Normally there's a steady stream of new connections and connection drops.<br>
<br>
I've listed the open file handles from the rabbitmq beam instance and I still see live file handles for the log files.<br>
<br>
</div>
Anybody have any idea what might be happening? And better still, how we might kick things without restarting the broker?<br>
<br>
</div>
Thanks,<br>
<br>
Matt<br>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<font face="Times New Roman" size="3"><span style="font-size:12pt" lang="EN-US">
<p class="MsoNormal" style="margin:0cm 0cm 0pt 36pt" align="left"><font face="Arial"><span style="font-size:10pt;font-family:Arial">To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
<br>
<a href="https://www.fnb.co.za/disclaimer.html" target="_blank">https://www.fnb.co.za/disclaimer.html</a>
<br>
<br>
</span></font></p>
<u></u><u></u></span></font><font face="Arial"><span style="font-size:10pt;font-family:Arial">
<p class="MsoNormal" style="margin:0cm 0cm 0pt 36pt"><font face="Arial"><span style="font-size:10pt;font-family:Arial">If you are unable to access the Disclaimer, send a blank e-mail to
</span></font><font face="Arial"><span style="font-size:10pt;font-family:Arial"><a href="mailto:firstrandbankdisclaimer@fnb.co.za" target="_blank">firstrandbankdisclaimer@fnb.co.za</a> and we will send you a copy of the Disclaimer.<u></u><u></u></span></font></p>
<p class="MsoNormal" style="margin:0cm 0cm 0pt 36pt" align="left"></p>
</span></font></div>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<font face="Times New Roman" size="3"><span style="FONT-SIZE:12pt" lang="EN-US">
<p class="MsoNormal" style="MARGIN:0cm 0cm 0pt 36pt" align="left"><font face="Arial"><span style="FONT-SIZE:10pt;FONT-FAMILY:Arial">To read FirstRand
Bank's Disclaimer for this email click on the following address or copy into
your Internet browser: <br><a href="https://www.fnb.co.za/disclaimer.html" target="_blank">https://www.fnb.co.za/disclaimer.html</a>
<br><br></span></font></p><u></u><u></u></span></font><font face="Arial"><span style="FONT-SIZE:10pt;FONT-FAMILY:Arial">
<p class="MsoNormal" style="MARGIN:0cm 0cm 0pt 36pt"><font face="Arial"><span style="FONT-SIZE:10pt;FONT-FAMILY:Arial">If you are unable to
access the Disclaimer, send a blank e-mail to </span></font><font face="Arial"><span style="FONT-SIZE:10pt;FONT-FAMILY:Arial"><a href="mailto:firstrandbankdisclaimer@fnb.co.za" target="_blank">firstrandbankdisclaimer@fnb.co.za</a>
and we will send you a copy of the Disclaimer.<u></u><u></u></span></font></p>
<p class="MsoNormal" style="MARGIN:0cm 0cm 0pt 36pt" align="left"></p></span></font>
</div></div></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div></div></div></div></div>