[rabbitmq-discuss] Determining Cause of Flow-Control Invocation

[Richard Raseley]

We currently use RabbitMQ as a broker for a subset of our overall logging
platform. Recently, we attempted to add an additional category of logging
that would result in additional load of approximately 2,000 msg/sec with an
average message size of approximately 2KB.

Upon adding this additional category of logging, we noticed a couple things:

(1) Our consumers were not able to keep up with the flow of messages, so
they began to build up in the queue (our developers are working on fixing
this).

(2) We began to see RabbitMQ invoke flow-control in the form of TCP
back-pressure on our producers.

My initial thought was that since we were using durable queues with
persistent messages that when the consumers weren't immediately picking up
the messages, RabbitMQ was attempting to persist them to disk in order to
satisfy the persistence & durability requirements and it couldn't keep up.

But in following up with my developers this morning it seems that they are
in fact no publishing the messages as persistent and being that (based on
my understanding) a non-persistent message published to a durable queue is
not written to disk, this threw my initial theory out the windows (I still
have to verify their claim of non-persistence).

So, my question is - what other things can I look at to determine the cause
of the invocation of flow control. Additional relevant information is as
follows:

(*) 2x RabbitMQ Nodes Running 3.1.3 in a clustered configuration.
(*) Each node has 8x 2GHzCPU cores and 4GB of RAM.
(*) At no time during the window in question did CPU utilization reach >15%
(*) At no time during the window in question did memory utilization reach
>50%
(*) In looking at the logs during the window in question I can see events
which I think are associated with the various publishers disconnecting (I
am not sure if our producer code is handling flow-control gracefully) but
nothing that seems to indicate why this happened.

Any help would be greatly appreciated.

Regards,

Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/4bd978e7/attachment.htm>