[rabbitmq-discuss] problem with new persiter with 1000 topics/queues

[alex chen]

Matthias,


> have you had a chance to run your tests against the latest (2.1.1) broker release? We fixed an excessive queue memory usage bug > in that release, which might well have been the cause of the problems you were seeing.

Thanks for reminding me!
I tested 2.1.1 release today with 1000 queues.  When the message store reached 120 GB, the broker's memory reached 4GB and started to slow down the publishers.  Publish rate dropped from 80 MB/sec to less than 1 MB/sec.  The memory stayed around 4GB for long time.  If I stopped the publishers, the memory would drop to 2.8 GB.  However, as soon as i started the publishers again, memory would reach 4GB again and the publish rate dropped significantly.

I tried to change the memory threshold from 0.4 to 0.7, and restarted the broker.  Howerver, the log showed that the broker's memory limit is 3.2 GB (our machine has 8 GB   RAM).  It seems to me release 2.1.1 would not allow the broker to use more than 0.4 of the total RAM.  I remember with 2.0.0, I could set the memory threshold to 0.7 and the broker's memory would reach 10 GB with 120 GB message stored.  the publish rate was higher than 2.1.1 under this condition.

BTW, we are using erlang R14A, because it uses 200% of the CPU (8 core) comparing to 680% for R13B with our test cases.
the 2.1.1 release has build error from rabbitmq_ssl.erl.  I commented out "-include(ssl/src/ssl_int.hrl)" to make it work.
i will test R14B tomorrow.

thanks.

-alex


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101026/64255b78/attachment-0001.htm>