[rabbitmq-discuss] problem with new persiter with 1000 topics/queues

[alex chen]

> Hmm, that's unfortunate. Really, you should only be bound by disk  write

> speed at that point, and I'm sure your disks can do more than  10MB/sec.
> You might try tuning the msg_store_file_size_limit setting: the  default
> is 16777216 (i.e. 16MB), but try pushing that up to 67108864 (i.e.  64MB)
> and you might find Rabbit can drive the disks harder. You'll need to  do
> that in rabbitmq.config with something like:
> 
> [{rabbit,  [{msg_store_file_size_limit, 67108864}]}].

The 10MB/sec was due to the broker memory passed 0.4 mem watermark and started 
putting throttling the producer.
The problem is after the producer slowed down, the broker still could not 
recover from the high memory usage.
After i increased the watermark to 0.7, the publish rate could reach 80 MB/sec.  


> We can reproduce that  here. I have a few theories about this, but it's
> really pending more testing  and debugging. Is there any chance you could
> send us your test code - it'd be  good to see exactly what you're doing?

our code is based on the amqp_consumer.c from the rabbitmq-c/examples, with some 
modifications for durable exchange/queues.
however, it is mixed with lots of our internal functions.  i will try to make 
minimum modification to amqp_consumer.c to reproduce our test case, and send you 
that sample code.

> Mmm. Try seeing what happens if you reduce  the high watermark to 0.2 or
> lower - yes, publishing will be slower, but it  might help with the
> memory usage on consuming.

i will try that.

> We're not quite sure yet either I'm afraid. As I  say, I have a few
> theories I want to test out but it could be an issue with  the Erlang
> garbage collector. Also, what version of Erlang are you  using?

good point!  i forgot to mention that i tried both erlang 13B and 14A.
with 13B, the broker would use very high cpu (>700% on our 8 core machine) while 
consuming 200 GB (even for 100 clusters), and the consuming rate would decline 
steadily during the test.
with 14A, the cpu stayed cool (about 200%) during both the 100 and 1000 clusters 
test.  the only problem is the mem usage.

thanks for looking into this!  

-alex