[rabbitmq-discuss] Messages disappearing from queues when clustered brokers are restarted.

[Andreas Lundberg]

Hi.

We have a problem where messages are disappearing from queues when
broker is restarted.

Case:
We running a test scenario of RabbitMQ 2.7 with several queues, where
we have two brokers (nodes) in cluster running on two different
RabbitMQ servers on two different machines.
We are testing with durable queues and persistent messages where we
execute the following procedure on the brokers:

1. Declare multiple queues and populate them with several messages
2. Restart the "master" broker(A) (and wait to the other broker (B)
becomes master)
3. Restart broker(B) and wait to broker(A) becomes master

Problem:

Wen we restart the "master" broker(A) everything behave as it should
and the other broker(B) becomes master,
but when we restart broker B(the new master) every message in every
queue disappears, but the brokers seems to run as normal.

Is there any way to prevent this kind of message loss, and why does it
happen?

Cluster setup:

* We follow the Clustering guide @ http://www.rabbitmq.com/clustering.html
to set up the cluster.
* Both brokers in clusters is set up to be disk nodes.

1. Setting up both machines with same cookie
2. Executing rabbitmq -detached to create independent broker on both
machines.
3. Execute rabbitmqctl cluster rabbit at brokerB on machine A
4. Execute rabbitmqctl cluster rabbit at brokerA rabbit at brokerB on
machine A (To make sure both brokers are disc nodes)

This results in two brokers(A and B) in a cluster, where both brokers
are disc nodes.

Executing rabbitmqctl cluster_status returns the following:
[{nodes,[{disc,[rabbit at brokerA,'rabbit at brokerB']}]},
 {running_nodes,[rabbit at brokerA,'rabbit at brokerB']}]


Queue setup:

We call declare queue (queueDeclare) with durable = true and RabbitMQ
properties containing x-ha-policy set to "all"
We also call publish (basicPublish) with PERSISTENT_BASIC
(deliveryMode 2 (persistent)).
We use publish confirms instead of transactions.



Hints and suggestions are highly appreciated.