[rabbitmq-discuss] Weird Issue when cluster member restarts

[Matthew Sackman]

Hi Arun,

On Fri, Nov 06, 2009 at 07:58:09AM +0530, Arun Suresh wrote:
> My setup is as follows :
> * 2 RabbitMQ (1.7.0 running on R13B02.1) servers in a cluster (say rabbit at m1,
> rabbit at m2).. both disk nodes.
> * 2 Clients (client at m1 connected to rabbit at m1, client at m2 connected to
> rabbit at m2). I am using the Erlang client (which currently has an issue when
> connecting to a server which is part of a cluster... this I fixed by setting
> "insist = true" in the #'connection.open' record)

Can you elaborate on this issue a bit? Is this redirection coming back
from the server?

> * both clients have processes that have subscribed to the same durable Queue
> (Q) using the same key (K). Q is bound to a durable topic exchange (X). X
> and Q are created on rabbit at m1.

I don't understand what you mean by a queue with a key. I take it you
have one topic exchange, one queue bound to the exchange with one
binding key (is this the K you refer to above?), and two consumers on
the queue. (From reading the rest of your email, yes this is clearly
what you mean).

> Now this is what i do..
> * I bring down rabbit at m1, and client at m1 (They are co hosted... so both are
> brought down together). I do a "rabbitmqctl list_queues" on rabbit at m2... I
> do not see Q there..

Ok, so the queue has been created on m1, not m2.

> * I bring rabbit at m1 up again... After which I see Q when i do a "rabbitmqctl
> list_queues".
> * Before I bring up client at m2, I publish a messge to X with a routing key
> K...
> Issue  : Since client at m2 is up and has a process subscribed to Q, I expect
> that process to receive the message... It doesnt..

Indeed it won't.

Clustering makes all exchanges and queues available on all nodes of the
cluster. *However*, a queue is still located on a single node, even
though it is accessible via all nodes. The queue will be created on the
node the client is connected to when it creates the queue. Thus your
queue has been created on m1, thus it's accessible from both m1 and m2,
but when m1 goes down, the queue will go down with it.

>From that point on, m2 has no record of the queue, so msgs sent to X
with K to go to Q will not ever reach Q - as far as m2 is concerned, Q
just does not exist.

When m1 comes back up, Q will be recovered.

> Further more... I notice that when i bring client at m2 back up, It restarts
> its subscribers... now all messages published to X with K are sent ONLY to
> client at m2.

Well client at m1 died. This means its subscriptions to Q were lost - there
is no concept of persisting subscriptions, so when rabbit at m1 comes back
up, the Q will have no consumers at all. If you restart client at m2 then
yes, it'll recreate its subscription to Q, and thus receive all the msgs
sent to that Q. If you also restart client at m1 then that too will
recreate its subscription to Q and share the msgs with client at m2 as
before.

Hope this is of help.

Matthew