[rabbitmq-discuss] Publishing Performance

[Simon MacMullen]

On 04/02/2014 7:30AM, Michael Klishin wrote:
> On 4 Feb 2014, at 11:24, Umutcan <umutcan at gamegos.com> wrote:
>
>> Does publishing get slower while number of the queued messages are increasing?
>>
>> While I am doing an operation, publishing speed get slower in time. Operation publishes a certain amount of messages to RMQ and then  a consumer processes messages. Since publishing speed is higher than consuming speed, messages waits for a while in the queue. I am not running operation again until queue is empty.
>
> If consumers do not keep up with publishers, eventually RabbitMQ will throttle publishers
> until consumers drain enough messages to eliminate RAM and disk space pressure.

First of all, RabbitMQ will start paging messages out to disk. Since at 
all times RabbitMQ tries not to accept messages faster than it can do 
something sensible with them, this will mean the publishing rate goes 
down substantially at this point.

You can see if a queue is paging to disk in the management plugin web 
UI, or by looking at backing_queue_status.ram_msg_count in rabbitmqctl 
list_queues or the HTTP API.

Eventually if the disk fills up, or something else happens to make 
memory usage spike, then the memory alarm can go off and block 
publishers completely until memory use drops again.

> See http://rabbitmq.com/memory.html.
>
> OS swapping can also take place, which affects performance severely.

We try to avoid that, but on a machine where other services are running 
it's not always possible.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal