[rabbitmq-discuss] RabbitMQ failover and message loss

[Matthew Sackman]

On Fri, May 21, 2010 at 06:55:57PM +0530, Radha Krishnan D wrote:
> I have 2 brokers clustered with each other.  They are load balanced by
> having a HA proxy before them.

Ok, that's not a usual setup. If you actually have the two rabbit nodes
clustered together then you should have no need for the load balancer -
all clients can connect to either node and still reach every resource in
the broker.

> Now both P1 and C1 get connected to B2 and the test continues till we
> stop all three.
> 
> Now, during the failover time we find message losses.
> 
> Is it expected  or can we make sure that no message gets lost.

That is expected. Queues are located on the node of the connection which
created the queue. When the node goes down, you'll lose that queue. When
your clients (re)connect to the broker and reach the other node, they
are recreating the queues and other resources, but are creating fresh,
empty queues at that point.

If you need to be able to withstand node failures then currently your
best bet is to use the Pacemaker Active/Passive guide at:
http://www.rabbitmq.com/pacemaker.html
You will need to publish all messages persistent to durable queues. You
should not need the load balancer and pacemaker will correctly ensure
that the two nodes are not both up at the same time.

> Also, when will be  channel.setReturnListener  called ? 

Well, you set it yourself. The ReturnListener is invoked as soon as any
Basic.Return method is read off the socket.

> Will it be called when the message does not reach the queue from the
> exchange or when the message does reach the consumer from the queue ?

This is the difference between immediate and mandatory. Mandatory says
"blow up if the msg doesn't make it to a queue". Immediate says "blow up
if the msg doesn't make it through a queue to a consumer".

> In the clustered mode, exchanges are global across rabbitmq nodes .. are
> the queues also global ?

They are visible globally within the cluster, but they are located in a
node.

Matthew