[rabbitmq-discuss] RabbitMQ periodic slowness

[Ron Cordell]

Hello everyone!

We are trying to understand RabbitMQ behavior that we see in a highly
available cluster and I'm hoping someone here can shed some light on it.

We have a 3-node cluster that is exposed to the application via an F5 load
balancer by a virtual IP. The majority of the queues that are created are
highly available queues in order to prevent message loss from happening in
the event of a catastrophic failure of a node.

The rabbit instances/nodes are 3.2.2 with Erlang 16B02.

The application consists of a series of REST HTTP interfaces which place
the incoming messages onto the rabbit queues.

When we load test the application we are seeing a periodic delay in writing
a message to Rabbit. The delay happens about once every few minutes and
lasts up to 30 seconds.

We have tested in several scenarios:
3 node load balanced cluster
single node, non ha-policy applied machine
on Windows Server 2008R2
on Linux (Oracle Linux)
on VMWare virtual machines on high speed SAN storage
on physical machines with SSD
direct connection with no LB to single node

We haven't tested every combination in this list but we've tried to isolate
I/O, operating system, machine characteristics, etc.

But in every test scenario under load we start to see the socket opened to
Rabbit on a write operation and no response for 20-30 seconds (with network
tracing tools).

Has anyone seen any behavior like this?

Our concern is that under load some REST interfaces will show periodic
slowness; we have SLAs of ~1 second on the interfaces.

Thanks for any input!

Cheers,

Ron Cordell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140206/a51f9f70/attachment.html>