[rabbitmq-discuss] RabbitMQ 3.0 Policy ha-all delete issue

[Matthias Radestock]

Mark,

On 21/11/12 14:11, Mark Ward wrote:
> The size of the message may not be exactly the problem but more like how
> large of a message, queue size, client count can the rabbitMQ cluster manage
> before breaking.  That is probably stating the obvious. It seems even with
> the RabbitMQ's high watermark and flow control I was still able to push
> rabbitMQ above the highwater mark and enough that erlang would crash and
> bring down rabbitmq.  The other was the erlang's heartbeat.  It would
> timeout if the cluster nodes could not exchange messages before the node
> timeout happened.
>
> The lower the high watermark the larger the message rabbitMQ was able to
> manage. But that also greatly increased the use of flow control.

The default setting of the HWM is designed to work reasonably well on a 
range of typical setups. But it is tuneable for a reason. In particular, 
highly memory constrained set ups, like the one you have, may need 
lowering of the watermark.

As for the erlang inter-node heartbeat... I recall there was an issue in 
Erlang where large messages between nodes would cause heartbeats to be 
missed. I thought that had been fixed but I might well be imagining it. 
The erlang heartbeat is tuneable though, so on overloaded systems and 
when message payloads are large it might be worth increasing the timeout 
value.

> My test cluster is pretty weak.  I have VMWare vSphere 5 hypervisor
> installed on an Intel Core 2 Duo 3.16GHz only 4 gig of ram.  I have 3 VMs
> running on this poor workstation with each VM being Windows 2008 r2.  Each
> VM only has 1 gig of ram given to it.

I wonder how much memory is actually left to applications in this setup, 
especially if you haven't got any swap space configured. The default HMW 
setting assumes that rabbit can take (nearly) all the available memory, 
which may be far too optimistic in this instance.

Regards,

Matthias.