[rabbitmq-discuss] Rabbit slowing down on accepting messages

[Ron Cordell]

Hi,

During our performance testing yesterday we saw some interesting behavior.
The setup is:

RabbitMQ x 3 nodes, clustered, behind a load balancer, running on Linux,
Erlang 16B03, Rabbit 3.2.3. RabbitMQ nodes running on VMWare ESX 4.5

What we observed was a run in the morning publishing and consuming a
sustained rate of 140K messages/hour across 150 mirrored queues (good).
The application was shut down, all connections to Rabbit were observed to
drop to 0.
An hour or two passes while the team does other things.
The application was brought back up, connections re-established, and
another test run kicked off.
This time the max sustained rate is about 65K messages/hour (bad). The
reason was narrowed down to Rabbit not accepting published messages any
faster than that.
Disk I/O was checked - we're on a very fast SAN and there was no issue.
Load balancer was checked, it's not preventing the issue.
Finally, the RabbitMQ nodes were stopped and restarted and another test run
was made, and suddenly we are back up to 140K messages/hour.
The logs show no memory alarms or flow control messages - in fact the logs
are "clean".

Any suggestions on places to look to see what the underlying issue might be?

In real life I don't expect to start and stop applications, so this may or
may not be a concern, but we are very curious about the observed behavior.

Cheers,

Ron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140319/c53afac5/attachment.html>