[rabbitmq-discuss] problem with new persiter with 1000 topics/queues
chen650 at yahoo.com
Tue Aug 17 08:21:41 BST 2010
Thanks to the fix you put in to the default branch, our test showed that it was
able to consume 10 GB of messages without any errors.
however, there is unexpected memory usage when consuming from 200 GB of backed
first, when i tried to publish to 1000 topics/queues, when the queue size
reached 100 GB, the memory usage would grow to 4GB. our system has 8 GB of
RAM. with the default high watermark of 0.4, the broker started to throttle the
publisher. the publish rate would drop from 80 MB/sec to 10 MB/sec. when queue
size reached 200 GB, i started the consumer, then the memory usage would grow
to 10 GB. the system started swapping and the consume rate dropped from 30
MB/sec to < 10 MB/sec.
later i tried to increase the high watermark to 0.7. the max mem usage for
broker reached around 6.6 GB while publishing to 200 GB. for some reason the
throttling did not occur, so the publish rate maintained at 80 MB/sec. however,
when starting to consume the messages from 200 GB, the mem usage would still
reach 10 GB. the system started swapping, the consume rate dropped from 30
MB/sec to 20 MB/sec. it is better than the 0.4 watermark case.
these tests were run with 1000 topics/queues. if i decrease the topics to 100,.
the broker's mem usage would remain below 3GB, even when the data size reached
any idea on how to decrease the usage of broker for the 1000 topics case? i am
not sure if it is caused by a bug, or it is something that cannot be avoided.
----- Original Message ----
> From: alex chen <chen650 at yahoo.com>
> To: Matthew Sackman <matthew at rabbitmq.com>;
rabbitmq-discuss at lists.rabbitmq.com
> Sent: Mon, August 16, 2010 10:42:21 AM
> Subject: Re: [rabbitmq-discuss] problem with new persiter with 1000
> Matthew and Matthias,
> Thanks a lot for fixing it so quickly! It is greatly appreciated!
> I will run some tests today using the default branch. will let you know the
> results asap.
More information about the rabbitmq-discuss