[rabbitmq-discuss] [help] [beginner] server stops sending messages; publish (in transaction) hangs on commit

[Marek Majkowski]

On Thu, Feb 23, 2012 at 02:47, Alistair Bayley <alistair at abayley.org> wrote:
> I'm looking for ways to debug a rabbitmq server that has been running
> since 2010-02-01. Sometime in the last 2 days it stopped delivering
> messages. I've seen this happen before, and restarting fixes it, but
> I'd like to get to the bottom of why it stops delivering messages, so
> I'd prefer to run tests while it is still in this state.
>
> We're using the python amqplib client. The various versions are:
>
> $ uname -srvmpio
> Linux 2.6.32-37-generic-pae #81-Ubuntu SMP Fri Dec 2 22:24:22 UTC 2011
> i686 unknown unknown GNU/Linux
>
> $ sudo rabbitmqctl report
> Reporting server status on {{2012,2,23},{2,18,37}}
>
> Status of node 'rabbit at rabbitmq1' ...
> [{pid,11649},
>  {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
>                        {os_mon,"CPO  CXC 138 46","2.2.4"},
>                        {sasl,"SASL  CXC 138 11","2.1.8"},
>                        {mnesia,"MNESIA  CXC 138 12","4.4.12"},
>                        {stdlib,"ERTS  CXC 138 10","1.16.4"},
>                        {kernel,"ERTS  CXC 138 10","2.13.4"}]},
>  {os,{unix,linux}},
>  {erlang_version,"Erlang R13B03 (erts-5.7.4) [source] [rq:1]
> [async-threads:30] [hipe] [kernel-poll:true]\n"},
>  {memory,[{total,107054896},
>          {processes,72060236},
>          {processes_used,72055268},
>          {system,34994660},
>          {atom,736077},
>          {atom_used,727538},
>          {binary,15125896},
>          {code,6063325},
>          {ets,10643172}]},
>  {vm_memory_high_watermark,0.3999999992195367},
>  {vm_memory_limit,102503219}]

Hi,

You seem to have quite a small amount of RAM - 100MiB, right?

Do logs show high memory watermark being set?

If so: rabbitmq tries not to go above a memory limit.
When rabbit hits memory limit it stops consuming messages -
all connections that produce messages will be stopped.
The connection won't be touched until some memory is freed.

But in setups where total memory is small, RabbitMQ may
enter a state of just being above memory watermark and not
being able to free anything. This may be happening to you.

Cheers,
   Marek