[rabbitmq-discuss] Confusing disk free space limit warning

Mon Sep 17 05:31:22 BST 2012

Hi guys,

I'm new to the list so please forgive me if I forgot any important info. 
First of all thanks to the rabbitmq devs for releasing a great product :)

I'm running Rabbitmq 2.8.4 and connecting to it with a celery / kombu / 
amqplib client. I'm running on Ubuntu 12.04.1 LTS on an Amazon EC2 
instance. After running for quite some time without issue, I saw this 
warning in the Rabbit log:

=INFO REPORT==== 16-Sep-2012::06:49:17 ===

=INFO REPORT==== 16-Sep-2012::06:49:17 ===
Disk free space limit now below limit. Free bytes:2000396288 
Limit:1000000000

=INFO REPORT==== 16-Sep-2012::06:49:17 ===
     alarm_handler: {clear,{resource_limit,disk,rabbit at mq1}}

The amount of free space reported in the log is around twice the amount 
of space that rabbit requires in this instance, so I can't understand 
why this log occurred. Following this rabbit did seem to rate limit all 
clients, so I believe that it was acting as though we had breached the 
limit. There were no subsequent rabbit logs for the next ~24 hours. Not 
knowing much about rate limiting, etc, we actually restarted our amqp 
clients and then rabbit itself, after which everything seemed to start 
working fine again.

This is the output of rabbitmq status (done on the next day, which was 
when i realised there was a throttling issue):

Status of node rabbit at mq1 ...
[{pid,828},
  {running_applications,
      [{rabbitmq_management,"RabbitMQ Management Console","2.8.4"},
       {xmerl,"XML parser","1.2.10"},
       {rabbitmq_management_agent,"RabbitMQ Management Agent","2.8.4"},
       {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.8.4"},
       {webmachine,"webmachine","1.7.0-rmq2.8.4-hg"},
       {mochiweb,"MochiMedia Web Server","1.3-rmq2.8.4-git"},
       {inets,"INETS  CXC 138 49","5.7.1"},
       {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","2.8.4"},
       {rabbit,"RabbitMQ","2.8.4"},
       {mnesia,"MNESIA  CXC 138 12","4.5"},
       {os_mon,"CPO  CXC 138 46","2.2.7"},
       {sasl,"SASL  CXC 138 11","2.1.10"},
       {amqp_client,"RabbitMQ AMQP Client","2.8.4"},
       {stdlib,"ERTS  CXC 138 10","1.17.5"},
       {kernel,"ERTS  CXC 138 10","2.14.5"}]},
  {os,{unix,linux}},
  {erlang_version,
      "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] 
[async-threads:30] [kernel-poll:true]\n"},
  {memory,
      [{total,35645328},
       {processes,12347544},
       {processes_used,12333872},
       {system,23297784},
       {atom,1548841},
       {atom_used,1538490},
       {binary,301336},
       {code,17536634},
       {ets,2116424}]},
  {vm_memory_high_watermark,0.3999999998873157},
  {vm_memory_limit,709948211},
  {disk_free_limit,1000000000},
  {disk_free,1808924672},
  {file_descriptors,
[{total_limit,924},{total_used,10},{sockets_limit,829},{sockets_used,6}]},
  {processes,[{limit,1048576},{used,225}]},
  {run_queue,0},
  {uptime,8035}]
...done.

Any help you guys could give with tracking down the cause of this issue 
would be appreciated.

Cheers,
Mark