[rabbitmq-discuss] major failure when clearing large queue backlog

Tue Aug 16 13:45:27 BST 2011

Since restarting rabbit is a major disruption to service, changing
that value will probably wait until next week at the earliest. Seeing
as I have some time, I'd like to make sure we set it to a value that
we're happy with.

The documentation says only that disabling memsup will prevent
throttling of producers. There's also discussion about the Erlang
garbage collector, which I can see being responsible for the situation
I originally described.

http://www.rabbitmq.com/extensions.html#memsup

We've been running this configuration for a long time now, and I
vaguely recall that when the spool-to-disk feature was added, it was
dependent on this setting. However, I've seen it write to disk even
with this setting off so I may be mistaken. Regardless, we don't have
support in our applications for flow control (yet?) so I've been wary
to enable vm_memory_high_watermark. I plan to add a concept of flow
control strategies to haigha so that common use cases can be
abstracted without per-app implementations.

I've seen rabbit use much more than 1GB so I don't think it's defaulted to that.

-Aaron

On Tue, Aug 16, 2011 at 6:35 AM, Matthew Sackman <matthew at rabbitmq.com> wrote:
> On Mon, Aug 15, 2011 at 01:29:23PM -0400, Aaron Westendorf wrote:
>> I can email you logs, but they're really not exciting. It's all just noise
>> as messages were published and consumers had yet to initialize and declare
>> exchanges, queues and bindings.
>
> Hmm, that's a pity.
>
>> I assume you have tests for spooling to disk when rabbit is low on memory.
>> Has this been tested in a cluster situation such as I described?
>
> Part of our test suite certainly does simulate low memory conditions.
>
>> I checked our configurations, and I wonder if this is what caused the
>> problem:
>> [{rabbit, [{vm_memory_high_watermark, 0}, {cluster_nodes,
>> ['artemis','hermes']}]}].
>
> Erm, setting that to 0 disables the memory monitor, which then forces
> Rabbit to default to thinking that there's probably about 1GB RAM
> installed. This is very certainly not what you want. I'd set that back
> to the default of 0.4 and see how you get on with that.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

-- 
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com