[rabbitmq-discuss] |Spam| MQ Cluster Replication Traffic Questions

[Simon MacMullen]

On 27/02/2014 7:47PM, Chris Larsen wrote:
> Despite the cross-over we were seeing network partition alerts with
> version 3.2.2. We saw NIC reset errors (Intel NICs) and just upgraded
> the drivers to fend off that problem and tried some buffer tuning. But
> we’re still dropping packets on the cross-over interface so I’m worried
> the partitions may continue. Here are the questions I have:
>
> 1)Is it a bad idea to use a cross-over like this?

It's certainly reasonable to use a separate network interface for 
clustering. But the network needs to be reliable - if you are seeing the 
NIC reset then that's probably not the case :-(

> 2)We’re seeing ~2.5Mbps in / ~10Mbps out on the public eth0 interface
> but ~45Mbps in / ~30Mbps out on the cross-over. Is that kind of
> amplification normal?

Depending on message size / use of mirroring, it could be. Mirroring 
increases the amount of traffic by a constant factor, and messages being 
sent across the cluster are somewhat more decoded than they are as AMQP, 
so can have a higher overhead.

> 3)If it’s ok to use the cross-over, what TCP tuning am I missing?

I'm afraid I can't offer advice on how to make your NIC reliable. But I 
would expect that TCP tuning parameters are not where the problem is 
likely to be.

Ultimately the occasional dropped packet should be fine; TCP should be 
able to recover from that! But if the NIC is resetting then you have a 
bigger problem.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal