[rabbitmq-discuss] flow control issues

Thu Sep 16 11:31:28 BST 2010

On Wed, Sep 15, 2010 at 14:45, romary.kremer at gmail.com
<romary.kremer at gmail.com> wrote:
> since I can not find out the reasons of our problems yet, I've decided to
> perform some more basics tests.

Hurray!

> To step in the memory limitations faster, I 've decreased the memory
> watermark threshold to .05 (5%)
> This corresponds to 50 MB on the host we are using for this test as shown on
> the rabbit.log file upon startup of broker :
> =INFO REPORT==== 15-Sep-2010::12:36:15 ===
> Memory limit set to 50MB.

Hooa. 50 megs is _really_ small. Probably more than half of it is
taken by Erlang
internals, really not much is left for Rabbit!
It's hard to say what is the minimal sensible memory allowance, but I'd guess
not less than 150 megs. Probably even 250 megs if you do anything non-trivial.

In your  50megs setup you may run into situation when once the memory
alarm goes on it _never_ goes off. That would mean you won't be able
to publish any message.

> Finally the broker status description (rabbitmqctl status)
> [{running_applications,[{rabbit_status,"RabbitMQ Status Page","0.01"},
>                         {rabbit,"RabbitMQ","2.0.0"},
>                         {ssl,"Erlang/OTP SSL application","3.10.7"},
>                          {public_key,"Public key infrastructure","0.4"},
>                          {mnesia,"MNESIA  CXC 138 12","4.4.12"},
>                        {os_mon,"CPO  CXC 138 46","2.2.4"},
>                          {sasl,"SASL  CXC 138 11","2.1.8"},
>                          {rabbit_mochiweb,"RabbitMQ Mochiweb
> Embedding","0.01"},
>                          {mochiweb,"MochiMedia Web Server","1.3"},
>                          {crypto,"CRYPTO version 1","1.6.3"},
>                          {inets,"INETS  CXC 138 49","5.2"},
>                          {stdlib,"ERTS  CXC 138 10","1.16.4"},
>                          {kernel,"ERTS  CXC 138 10","2.13.4"}]},
>  {nodes,[{disc,['rabbit at mystic-buntu']}]},
>   {running_nodes,['rabbit at mystic-buntu']}]

Ok. For testing and benchmarking I'd recommend to remove status plugin.
Status plugin (if you look at the results) constantly queries Queues, which
may in some cases make it less likely to GC/hibernate them.

The new management plugin is better and shouldn't affect the GC that much.
(though for benchmarking it's still better to disalbe all the pluigins)

> - The last lines in the broker log are :
> =INFO REPORT==== 15-Sep-2010::14:53:44 === vm_memory_high_watermark set.
> Memory used:53650536 allowed:52556595 =INFO REPORT==== 15-Sep-2010::14:53:44
> === alarm_handler: {set,{vm_memory_high_watermark,[]}}
> From that point, the producer is blocked without any possible recovery. As
> the flow control is designed in v 2.0.0, I would have expected the producer
> to be released thanks to sawp of messages to the disk.

Yes it will! But only if it has enough memory to breathe!

What has happened is that internal Erlang stuff took most of the memory,
and even if we write everything to disk we won't be able to keep the
memory usage
under the threshold.

> I think this behaviour is not expected and maybe that could be due to a bug
> somewhere, since it si fully reproductible on my configuration.
> The fact that the memory occupation never falls below the threshold after
> message are removed from the queue is particularly strange and
> unwilling from my point of view. It think that this simple test can points
> out an issue that would explains the problems that I mentioned in the
> previous messages, but not sure about it.
> I am sorry for this quite long message, but I thought that the more details
> you get, the better.

Few suggestions:
 - Give rabbit more memory. How much memory? That depends on your setup.
   If you run 10k of connections you need enough memory to handle those
   connections + some memory to keep messages and queues.
   If the connections will eat up all the memory - rabbit won't allow
you to publish.
 - Disable status plugin (you can try to use management plugin).
 - Try new Erlang, R14B. They claim to have fixed some memory leaks with SSL,
   so it may help.

Cheers!
  Marek