[rabbitmq-discuss] rabbitmq nodedown - nodedown - Generic server rabbit_disk_monitor terminating

[Michael Sander]

Hi Michael,

*Version: *I'm running debian linux on a Google Compute Engine instance
with RabbitMQ 3.1.3. (more version info below).

*Workload:* I'm using rabbitmq as part of a process to OCR many PDFs.  I
add many URLs to various PDFs into rabbitmq. Then, a consumer pulls that
link out of rabbitmq, downloads the file, attempts to OCR it, and then
sends the result to another server.

*Diskspace: *During OCR, my app writes, reads, and deletes a lot of
temporary files, so it is possible that I briefly an out of free disk space
momentarily. But I ran df afterwards and it looks like I have enough space.
 (see below). Even if I did run out of disk space, shouldn't rabbitmq be
somewhat graceful about it? Perhaps it should refuse any new jobs while
there is no space but then come back online once it detects there is space
available.

*Reproducing: *Unfortunately nothing reliably reproduces it.  This is one
of those annoying situations where everything is working fine for a week
and then all of a sudden it goes down. I'm going to setup a script that
will email me when my disk space goes over 90%, so hopefully that will help
identify the issue.

Here is some additional version and disk usage information that may be
useful

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           10G  6.9G  2.7G  73% /
/dev/root        10G  6.9G  2.7G  73% /
none            899M     0  899M   0% /dev
tmpfs           180M  4.1M  176M   3% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           360M     0  360M   0% /run/shm
$ cat /proc/version
Linux version 3.3.8-gcg-201305291443 (nobody at google.com) (gcc version
4.6.x-google 20111101 (prerelease)
(Google_crosstoolv15-gcc-4.6.x-glibc-2.11.1-grte) ) #1 SMP Wed May 29
14:49:59 PDT 2013

Appreciate the help.

Best,

Michael Sander
michael.sander at gmail.com
607-227-9859


On Tue, Oct 29, 2013 at 8:23 AM, Michael Klishin <mklishin at gopivotal.com>wrote:

> On 29 Oct 2013, at 08:56, Michael Sander <michael.sander at gmail.com> wrote:
>
> > ** Reason for termination ==
> > ** {{badmatch,[]},
> >     [{rabbit_disk_monitor,parse_free_unix,1,[]},
> >      {rabbit_disk_monitor,internal_update,1,[]},
> >      {rabbit_disk_monitor,handle_info,2,[]},
> >      {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},
> >      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
> > =INFO REPORT==== 28-Oct-2013::19:48:36 ===
> > Disabling disk free space monitoring on unsupported platform: {{'EXIT',
> >
> {{badmatch,[]},
> >
>  [{rabbit_disk_monitor,
> >
>  parse_free_unix,
> >                                                                    1,[]},
> >
> {rabbit_disk_monitor,
> >
>  init,1,[]},
> >
> {gen_server,
> >
>  init_it,6,
> >
>  [{file,
> >
>  "gen_server.erl"},
> >
> {line,
> >
>  304}]},
> >
> {proc_lib,
> >
>  init_p_do_apply,
> >                                                                    3,
> >
>  [{file,
> >
>  "proc_lib.erl"},
> >
> {line,
> >
>  227}]}]}},
> >
>  1887428608}
>
> Michael,
>
> Is there a more or less reliable way to reproduce the issue? E.g. what OS,
> RabbitMQ
> version and RabbitMQ configuration can we try? What is your workload like?
>
> Thank you.
>
> MK
>
> Software Engineer, Pivotal/RabbitMQ
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131029/cb5321ea/attachment.htm>