[rabbitmq-discuss] High publish rates, mirrored queues, high watermark...

[Jim Myhrberg]

Hi, I was hoping you guys would be able to answer some quick questions about an issue we're having.

We've recently started using mirrored queues across two RabbitMQ instances on two servers, and it's working really well except for when publish rates are really high for a longer period of time. At that point RabbitMQ starts eating a lot of memory on the master node until it hits our high watermark limit of 18.9GB, at which point things basically stop working.

My theory of what is happening is that when RabbitMQ can't process incoming publish requests fast enough, it caches the raw TCP packets in memory until it can deal with them and direct it to the queue(s) in question. And I'm assuming the fact that we are using mirrored queues adds overhead in dealing with a publish request as it needs to be synced to the other node(s). Am I right?

My theory is based on the fact that we could deal with much higher publish rates without and problems before we switched to mirrored queues. And the fact that we've had a few points when memory usage has been excessive we've seen queues that were completely empty according to the management plugin, but our workers were still processing +1000 messages/sec for a good hour, all the while the management plugin said no incoming messages, and +1000 msg/s get/acks.

Thanks :)


-jim 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120112/602e7f4d/attachment.htm>