[rabbitmq-discuss] Persister logs never getting rolled/cleaned up?

[Marek Majkowski]

Steve,

On Thu, Jan 6, 2011 at 03:43, Steve Garrity <sgarrity at gmail.com> wrote:
> Apologies if this is a n00b question, which I think it is, but we're
> running a plain vanilla install of rabbitmq 2.1.1 on erlang r14B with
> celeryd 2.1.4 and we're continually running into persister log issues
> and Google has thus far not turned up results.
>
> Every few weeks rabbit just stops accepting new connections...stopping
> and starting rabbit takes on the order of 15 minutes and it just
> continues to choke once it gets started again. Looking at /var/lib/
> rabbitmq/mnesia/rabbit shows ~170MB of persister logs (this is running
> on a 1GB RAM machine, so the vm memory alarm is set at ~400MB (which
> we routinely bounce off of trying to reload these logs). By deleting
> (or renaming) this directory, recreating users/vhosts, and restarting
> rabbit, everything works great again (for a few weeks).

Sounds scary. You definitely shouldn't need to go that far!

> This led us to believe we were using celery incorrectly and
> accidentally persisting results for all time, but we've tried with
> ignore_result=True, delivery_mode=transient, durable=False, and
> result_persistent=False and nothing seems to fix it.
>
> These don't seem like logs that we should have to rotate
> manually...can someone point us in the right direction?

Rabbit should do everything automatically, something is very
wrong if it doesn't.

Rabbit only stops "accepting" new connections when it's running
out of file descriptors - I'm sure you would have noticed that.
On the other hand Rabbit does "block" connections which
publish messages when it's running out-of-memory
(ie: grows above 400megs in your case).

You can easily detect such a situation by looking in the
rabbit logs, it will say something like:

=INFO REPORT==== 7-Jan-2011::10:11:16 ===
vm_memory_high_watermark set. Memory used:XXX allowed:XXX


I'd guess, that in your case the rabbit is probably just overloaded.
400 megs of RAM for Rabbit is plenty if you're not doing
anything complex. But once you have a nontrivial setup
or many messages, it may not be enough.

Can you please:
 - run rabbitmqctl list_queues or management plugin
   to see how many messages Rabbit is usually
   storing in your setup, to make sure it's not overloaded
   by a high number of messages.
   (here are details about management plugin:
   http://www.rabbitmq.com/management.html )
 - Take closer look at the result of 'rabbitmqctl list_queues
messages_unacknowledged'.
   If this number is growing, something really bad is happening.

Cheers,
  Marek