[rabbitmq-discuss] [help] [beginner] server stops sending messages; publish (in transaction) hangs on commit

[Marek Majkowski]

On Thu, Feb 23, 2012 at 16:58, Alistair Bayley <alistair at abayley.org> wrote:
> On 24 February 2012 01:01, Simon MacMullen <simon at rabbitmq.com> wrote:
>> On 23/02/12 11:34, Marek Majkowski wrote:
>>>
>>> You seem to have quite a small amount of RAM - 100MiB, right?
>>>
>>> Do logs show high memory watermark being set?
>>
>>
>> More to the point, the OP's "rabbitmqctl report" output shows memory use
>> above the watermark. So yes...
>>
>>
>>> If so: rabbitmq tries not to go above a memory limit.
>>> When rabbit hits memory limit it stops consuming messages -
>>
>>
>> As in, it will block on message publish. In tx mode that equates to a
>> commit.
>>
>>
>>> all connections that produce messages will be stopped.
>>> The connection won't be touched until some memory is freed.
>>>
>>> But in setups where total memory is small, RabbitMQ may
>>> enter a state of just being above memory watermark and not
>>> being able to free anything. This may be happening to you.
>>
>>
>> ...and you can set vm_memory_high_watermark higher to work around this. See
>> http://www.rabbitmq.com/memory.html.
>
> Thanks.
>
> Why is it not able to free memory? A rabbitmqctl report on a freshly
> started server shows total memory about 10M, what causes it to grow
> and not shrink back? This server is currently doing literally nothing
> - all queues are empty - so I'd like to know why total memory doesn't
> drop back down to something close to the initial state.
>
> It is running on a deliberately small VM. As it is the only service on
> the VM (the VM exists solely to host rabbitmq) we can increase the
> vm_memory_high_watermark . I would try 80%, is there a reason to go
> higher (or not)?.

Unfortunately Erlang memory handling is quite complex. By default,
Erlang tries to optimize what it can, and that may mean allocating more memory
that you would want to.

Basically - Erlang VM is quite heavy even if it is not doing much.

I'm afraid you just need to:
 a) buy more RAM
 b) or, tweak the vm_memory_high_watermark . You can even set
  it to be more than 100% - if you don't mind getting erlang
  process into swap sometimes.
  But in such case, it's better to make sure that you won't have much
  data residing in queues. (It should be okay to have high throughput
   through rabbit, as far as consumers are keeping up)

Marek