[rabbitmq-discuss] Enabling publisher confirms damages performance by a factor of 10x

[Matthias Radestock]

On 19/08/13 10:35, ModronMaze wrote:
> I tried to use publisher confirms to ensure my messages are not lost, but
> encountered a performance issue. My test case is as follows:
>
> 1. One publisher sends 200.000 messages in a loop and asynchronously marks
> them as confirmed.
> 2. One consumer reads messages from queue, executes some work (db access,
> sends other messages etc)
> 3. Prefetch is set to 512.
> 4. Queue and messages are durable/persistent.
> 5. I tried using versions 3.0.2, 3.1.4, 3.1.5
>
> Without using publisher confirms I publish at least *250 msg/s* (and ack
> with the same speed). Queue contains only prefetched messages.
>
> When I enable confirms (separate channel and connection - only for sending
> messages that require it) speed drops to *15-25 msg/s*. Using management API
> I found out that publisher's connection is constantly blocked by flow
> control.
> I think it is strange, because all messages are consumed as soon as they
> enter a queue.

The messages still need to be written to disk (modulo any internal 
optimisations) since the broker needs to take responsibility for them 
until they get acknowledged.

The performance you are seeing is what I'd expect when publishers wait 
for confirmation after every single message. But you say above that 
publishers mark messages as confirmed "asynchronously", so presumably 
you are not doing that.

Here's a test using MulticastMain (which ships with the Java client), 
which I believe reproduces your test case exactly, except that the 
consumer is not doing any work and the messages are 1k in size (you 
didn't indicate how large the messages are, so I just guessed).

$ runjava.sh com.rabbitmq.examps.MulticastMain -f persistent -c 1000 -q 
512 -C 200000 -s 1024 -i 10
starting consumer #0
starting producer #0
...
time: 50.493s, sent: 1999 msg/s, confirmed: 2000 msg/s, nacked: 0 msg/s, 
received: 1546 msg/s, min/avg/max latency: 9165385/11000052/11717636 
microseconds
...
time: 70.532s, sent: 1954 msg/s, confirmed: 1979 msg/s, nacked: 0 msg/s, 
received: 1559 msg/s, min/avg/max latency: 13392164/15317946/16613307 
microseconds
...

As you can see, this results in a sending rate of about 2kHz and a 
receiving rate of about 1.5kHz.

Running the same test with "synchronous" confirm, i.e. waiting for a 
confirm after every message ("-c 1"), drops the rate to about 10Hz.


Regards,

Matthias.