[rabbitmq-discuss] RabbitMQ 3.0.2 stops logging, but otherwise looks healthy?

[Jeffery, Mark]

Hello

You mentioned a cluster.
Is this happening on every box ?

Also, is the rabbit process running as "foobar" ?

Jeff

Matt Pietrek <mpietrek at skytap.com> wrote:



Here you go Mark. The file date/times line up with when I last saw logging in the files:

23:14 PROD mpietrek at foomq1:/proc/16050$ sudo ls -latr fd
total 0
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 2 -> /foobar/logs/foomq1.foo.bar.com/rabbitmq-server.log<http://foomq1.foo.bar.com/rabbitmq-server.log>
dr-xr-xr-x 7 foobar foobar  0 2013-06-02 11:50 ..
dr-x------ 2 foobar foobar  0 2013-06-02 11:50 .
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 9 -> pipe:[122339989]
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 8 -> /foobar/logs/foomq1.foo.bar.com/rabbit at foomq1sasl.log<http://foomq1.foo.bar.com/rabbit@foomq1sasl.log>
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 7 -> /foobar/logs/foomq1.foo.bar.com/rabbit at foomq1.log<http://foomq1.foo.bar.com/rabbit@foomq1.log>
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 6 -> socket:[122339982]
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 5 -> socket:[122339980]
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 4 -> pipe:[122339970]
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 3 -> pipe:[122339970]
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 17 -> socket:[122340310]
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 16 -> socket:[122340285]
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 15 -> /foobar/var/lib/rabbit at foomq1/msg_store_persistent/397.rdq
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 14 -> /foobar/var/lib/rabbit at foomq1/msg_store_transient/0.rdq
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 13 -> /foobar/var/lib/rabbit at foomq1/queues/D1KLLKOJLVJ4YZGYLP2D32A9W/journal.jif
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 12 -> pipe:[122339990]
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 11 -> /foobar/var/lib/rabbit at foomq1/LATEST.LOG
lrwx------ 1 foobar foobar 64 2013-06-02 11:50 10 -> socket:[122339997]
l-wx------ 1 foobar foobar 64 2013-06-02 11:50 1 -> /foobar/logs/foomq1.foo.bar.com/rabbitmq-server.log<http://foomq1.foo.bar.com/rabbitmq-server.log>
lr-x------ 1 foobar foobar 64 2013-06-02 11:50 0 -> /dev/null
lrwx------ 1 foobar foobar 64 2013-06-02 11:51 18 -> socket:[122340314]
l-wx------ 1 foobar foobar 64 2013-06-02 12:28 21 -> /foobar/var/lib/rabbit at foomq1/queues/ETWUA9HXDWBIXFTEF0DVH5FBZ/journal.jif
lr-x------ 1 foobar foobar 64 2013-06-02 12:28 19 -> /foobar/var/lib/rabbit at foomq1/queues/22TABTKBIJ6DQCA5GC6EZ0B8L/journal.jif
l-wx------ 1 foobar foobar 64 2013-06-02 13:12 22 -> /foobar/var/lib/rabbit at foomq1/queues/A9AWP930R556ERL81W9PYEELL/journal.jif
lrwx------ 1 foobar foobar 64 2013-06-02 13:12 20 -> /foobar/var/lib/rabbit at foomq1/queues/A8EPKFQ4VCO5A3APTTWYEQRFT/journal.jif


On Tue, Jul 16, 2013 at 12:19 PM, Jeffery, Mark <MJeffery at fnb.co.za<mailto:MJeffery at fnb.co.za>> wrote:

Hello

Assuming Linux, what does ls -latr show in the fd directory in the rabbitmq process's directory under /proc ?


Matt Pietrek <mpietrek at skytap.com<mailto:mpietrek at skytap.com>> wrote:



This is admittedly a shot in the dark, and with limited repro details, but asking if somebody else has seen this.

We run our clustered RabbitMQ 3.0.2 brokers in production without restarts for months at a time. Everything is looking good - No memory leaks, no out-of-disk-space, etc...

However, in looking at the rabbit@<node>.log and rabbit@<node>-sasl.log files, I noticed that at some point (e.g. a month earlier), no new entries were being added to the log file. Normally there's a steady stream of new connections and connection drops.

I've listed the open file handles from the rabbitmq beam instance and I still see live file handles for the log files.

Anybody have any idea what might be happening? And better still, how we might kick things without restarting the broker?

Thanks,

Matt

To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
https://www.fnb.co.za/disclaimer.html

If you are unable to access the Disclaimer, send a blank e-mail to firstrandbankdisclaimer at fnb.co.za<mailto:firstrandbankdisclaimer at fnb.co.za> and we will send you a copy of the Disclaimer.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser: 
https://www.fnb.co.za/disclaimer.html 

If you are unable to access the Disclaimer, send a blank e-mail to
firstrandbankdisclaimer at fnb.co.za and we will send you a copy of the Disclaimer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130717/cf7a4534/attachment.htm>