[rabbitmq-discuss] RabbitMQ queue out-rate settles to in-rate when many messages in queue

[Simon MacMullen]

On 12/05/2014 11:26, Jelle Smet wrote:
> Hi all,
>
> RabbitMQ: 3.3.1
> Erlang: 17
>
> We have following situation:
>
>   * A queue contains a large amount of messages (5 million because the
>     consumers are shutdown)
>   * There's a constant  incoming message flow at a rate of ~2000 msg/s.
>   * Messages are persistent though diskIO is far from being saturated.
>   * Consumers have a prefetch value of 1000
>   * We presume backpressure is applied to the producer in this situation
>     (difficult to observe).

Check mgmt / ctl - do the producer connections have a state of "flow" 
(backpressure is applied) or "running" (it's not)?

> What we observe is that out-rate settles to the in-rate, which I don't
> understand because:

When queues in RabbitMQ 3.3.x are the bottleneck, they will deliver (at 
least) 10% more messages than they accept:

http://www.rabbitmq.com/blog/2014/04/10/consumer-bias-in-rabbitmq-3-3/

[I know you say in-rate == out-rate but do you mean exactly, or to the 
nearest 10%?]

Note that previous versions of RabbitMQ did not make guarantees about 
this at all.

You can see whether your queues are the bottleneck as discussed here:

http://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/

>   * There a large amount of messages in the queue so plenty to consume.
>   * We KNOW consumers are able to write at a much higher throughput even
>     after bumping prefetch value.
>   * When the queue is empty, in & outgoing rate is again 8000 msg/s

When the queue is empty then you might be hitting an optimisation which 
can make persistent messaging much faster.

When a persistent message reaches a queue, it is also added to an 
internal queue to go to the disc. It will not be confirmed to the 
publisher until it has made it through this queue and on to the disc.

But we *can* deliver it to consumers before it has made it to disc! If a 
consumer receives it, and acknowledges it (or consumed in autoack mode), 
and that ack makes it back to the queue before the message has reached 
disc, then we cancel the write and send the confirm to the producer 
straight away, since RabbitMQ has already fulfilled all its 
responsibilities for that message by delivering it.

Obviously this optimisation only works when you have (very) fast 
consumers, and an empty or near-empty queue.

Could that explain the difference in performance you are seeing?

Cheers, Simon