[rabbitmq-discuss] 3.0.4 extremely unstable in production...?

[Simon MacMullen]

To add to what Emile said: the only difference between partition 
handling in 2.x and 3.x is that 3.x will show a big red warning in 
management when one has occurred whereas 2.x will stay silent. If you 
still have logs from the 2.x days you might want to grep for 
"running_partitioned_network" - I suspect you will find some matches.

The next release, 3.1 will have some features around automatic healing 
of network partitions.

Cheers, Simon

On 15/04/2013 10:16, Emile Joubert wrote:
>
> Hi,
>
> On 12/04/13 19:36, Matt Wise wrote:
>> Since creating the new server farm though we've had 3 outages. In the
>> first two outages we received a Network Partition Split, and effectively
>> all 3 of the systems decided to run their own queues independently of
>> the other servers. This was the first time we'd ever seen this failure,
>> ever. In the most recent failure we had 2 machines split off, and the
>> 3rd rabbitmq service effectively became unresponsive entirely.
>
> Versions 2.8.x and 3.0.x are equally susceptible to partitions. You can
> confirm this experimentally by setting up a cluster of v2.8.x nodes and
> interrupting connectivity for twice the net_ticktime (60s by default).
>
> See https://www.rabbitmq.com/partitions.html
>
>> Up until recently though I had felt extremely comfortable with
>> RabbitMQ's clustering technology and reliability... now ... not so much.
>> Has anyone else seen similar behaviors? Is it simply due to the fact
>> that we're running cross-zone now in Amazon, or is it more likely the 3
>> servers that caused the problem? Or the 3.0.x upgrade?
>
> A network outage coincided with the period when nodes were running
> v3.0.4. The network interruption is the cause of the partition rather
> than the broker version.
>
> At the time of writing RabbitMQ clustering does not tolerate network
> partitions well, so it should not be used over a WAN. The shovel or
> federation plugins are better solutions for that case.
>
> See http://www.rabbitmq.com/clustering.html
>
>
>
> -Emile
>
>
>
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>