[rabbitmq-discuss] problem with new persiter with 1000 topics/queues

[alex chen]

Matthew,

Thanks to the fix you put in to the default branch, our test showed that it was 
able  to consume 10 GB of messages without any errors.

however, there is unexpected memory usage when consuming from 200 GB of backed 
up data.
first, when i tried to publish to 1000 topics/queues, when the queue size 
reached 100 GB, the memory usage would grow to 4GB.  our system has 8 GB of 
RAM.  with the default high watermark of 0.4, the broker started to throttle the 
publisher.  the publish rate would drop from 80 MB/sec to 10 MB/sec.  when queue 
size reached 200 GB, i started the consumer,  then the memory usage would grow 
to 10 GB.  the system started swapping and the consume rate dropped from 30 
MB/sec to < 10 MB/sec.

later i tried to increase the high watermark to 0.7.  the max mem usage for 
broker reached around 6.6 GB while publishing to 200 GB.  for some reason the 
throttling did not occur, so the publish rate maintained at 80 MB/sec.  however, 
when starting to consume the messages from 200 GB, the mem usage would still 
reach 10 GB.  the system started swapping, the consume rate dropped from 30 
MB/sec to 20 MB/sec.  it is better than the 0.4 watermark case.

these tests were run with 1000 topics/queues.  if i decrease the topics to 100,. 
the broker's mem usage would remain below 3GB, even when the data size reached 
200 GB.

any idea on how to decrease the usage of broker for the 1000 topics case?  i am 
not sure if it is caused by a bug, or it is something that cannot be avoided.
thanks.

-alex




----- Original Message ----
> From: alex chen <chen650 at yahoo.com>
> To: Matthew Sackman <matthew at rabbitmq.com>; 
rabbitmq-discuss at lists.rabbitmq.com
> Sent: Mon, August 16, 2010 10:42:21 AM
> Subject: Re: [rabbitmq-discuss] problem with new persiter with 1000 
>topics/queues
> 
> Matthew and Matthias,
> 
> Thanks a lot for fixing it so quickly!   It is  greatly appreciated!
> I will run some tests today using the default  branch.  will let you know the 
> results  asap.
> 
> -alex