[rabbitmq-discuss] Mirror queues and poor write performance

[Matthew Sackman]

On Wed, Oct 26, 2011 at 02:10:46PM -0700, Anderew wrote:
> I have been doing a very simple test and have found that in a two node
> cluster write performance where the exchange is bound to a mirrored
> queue is amazingly slow (orders of magnitude slower) compared to a non-
> mirrored queue in an otherwise identical configuration. Dequeue
> performance appears unaffected (i.e. very fast). The queues and
> exchanges are non-durable in both cases. Both nodes in the cluster are
> on the same machine so network latency will not be an factor.

How are you measuring publishing performance? Publishing is asynchronous
so unless there's some sort of TCP backpressure going on (not impossible
at all), I wouldn't have expected publishing performance to be affected.
That said, it definitely is the case that much much more CPU is being
spent per message and thus if you watch the queue length grow, it'll
grow at a much slower rate than with a non-mirrored queue.

When publishing to a single plain queue on a remote node, the msg is
sent between the nodes (between the node to which you're connected, and
the node upon which the queue resides) once.

When publishing to a mirrored queue where the master is on the node to
which you're connected, the msg will make at least 3 inter-node hops,
and there are further msgs associated with each msg that'll have to go
between nodes. This work alone (serialisation and de) adds substantial
cost.

> What are the expectations for mirror queue write performance? I
> appreciate there is no such thing as a free lunch and that inevitably
> there would be a runtime cost for mirroring.

I can't remember what the last figures were last time I benchmarked.
I'll try and find time to do some quick tests tomorrow and report back.
But my recollection is that the cost is fairly substantial. I have made
some attempts to minimise this cost - batching in a few places goes on
to try and maximise gain from serialising fewer bigger structures rather
than loads of little ones, but there are probably further opportunities
for optimisations yet to be made. That said, inevitably, there is an
awful lot more work that has to get done per message.

Matthew