[rabbitmq-discuss] Publisher confirms performance

Tue Sep 11 21:40:20 BST 2012

Matthias,

Thanks for the response, I really appreciate it. I am running 2.8.x. Good
points on all the measuring too. I probably wasn't clear enough in my post,
but what I was saying was if I remove the confirms piece, and make the
queue non-durable, the speed of publishing a single message is < 1 ms. I'm
using Ruby and a rudimentary AMQP client that doesn't do any buffering.

Coupled with your explanations and a little bit of research, it sounds like
publisher confirms is only beneficial when done handled asynchronously. I
am trying to publish messages from a Ruby web server. Ruby's threading
model is very weak and uses in-process green threads. Because of this, I
was trying to avoid running any additional threads to perform the async
confirms.

It really sounds like with my implementation, I'm doing what RabbitMQ
transactions does with minor overhead because of an additional network
call. I don't think I had a great enough understanding of confirms when I
started this, it sounds like it's only beneficial over transactions if you
can receive the confirms asynchronously. Is that true? Thanks again for
your help.

On Mon, Sep 10, 2012 at 5:36 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> Allen,
>
>
> On 10/09/12 22:06, Allen Riddle wrote:
>
>> I am curious what people are seeing in their production deployments
>> of RabbitMQ using publisher confirms in a cluster of nodes with
>> durable queues.
>>
>
> Do you see the same results when running your tests against a single
> node instead of a cluster? I suspect so, but would be good to get
> confirmation of that.
>
> Also, I am assuming you are running a recent (2.8.x) version of rabbit,
> right?
>
>
>  With local benchmarks against a cluster with 2 nodes, I'm
>> synchronously publishing messages and confirming each, and it takes
>> roughly 46 ms on average per message.
>>
>
> You are asking RabbitMQ to confirm that the message has been written to
> disk. That requires an fsync. Which is very, very expensive.
>
> You can amortise that cost by: a) dealing with confirms asynchronously,
> rather than waiting for a confirmation after every publish, and b)
> publishing on multiple connections/channels, as long as that is
> compatible with the message ordering guarantees your app requires.
>
>
>  When I take away the publisher confirms piece, it falls down to 8 ms,
>>
>
> Be careful what you are measuring here. If you don't wait for confirms
> then publishing is an entirely asynchronous operation, so if you are simply
> timing the duration of the 'publish' API call then you may just be timing
> how long it takes for the client API to stuff the message into a local
> buffer.
>
> Anyway, taking away confirms means that the client won't have to wait for
> a round trip to the server, and the fsync delay there.
>
>
>  and when I remove durable queues, there is a net zero difference.
>>
>
> Not sure what you mean here. Zero difference compared to what? Making the
> queues non-durable eliminates the need to wait for an fsync. There's still
> the roundtrip latency though.
>
> Regards,
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120911/4a8144b6/attachment.htm>