[rabbitmq-discuss] problem with new persiter with 1000 topics/queues

[Matthias Radestock]

Alex,

alex chen wrote:
> So is there a plan to improve the memory usage in the new persister?

Over time the new persister will be improved in many ways. And memory 
use is definitely an area we will continue to look at. Quite when that 
will happen is impossible to predict.

> if not, we have to order new machines with 16 GB RAM instead of the
> current 8 GB.

Unless the costs are crippling I'd recommend that anyway.

> If we increase the file descriptor limit, would it reduce to memory usage?

yes, indirectly, by allowing rabbit to work more efficiently.

> i saw the following line in rabbit.log:
> "Limiting to approx 16284 file handles (14654 sockets)"
> is this too low?

That should be plenty.

> thanks a lot for all the inputs you provide on this problem.  that would help us 
> to figure out the system bottleneck and do the hardware planning accordingly.

Your problem reports have helped us iron out some bugs and sub-optimal 
behaviour in the new persister code. Thanks for testing the code in its 
pre-release form.

One thing that would help us identify potential bottlenecks in the 
broker is packaged test cases that demonstrate problematic behaviour. By 
"packaged" I mean test code that has been condensed (i.e. contains no 
extraneous code not related to the problem) and includes precise 
instructions on how to build and run it to reproduce the problematic 
behaviour and a clear description of what that behaviour is.

As you have seen yourself, altering just one little aspect of the test 
setup can produce radically different results. So for us to reproduce a 
problem we must ensure that we are running exactly the same code and 
follow exactly the same steps.

Would it be possible for you to package your tests in that way?


Regards,

Matthias.