[rabbitmq-discuss] Unable to join cluster: "Bad cookie in table definition rabbit_queue"

[Alexandru Scvorţov]

Hi,

As a rule, if you're breaking up a cluster completely, you need to reset
all nodes.  Otherwise, some nodes may still think they're in the cluster
and contact the nodes that aren't, leading to strange situations.

> (it was on nodes A,B,C,D and became B,C,D,E).
> I killed rabbitmq on nodes B,C,D (forgot to kill it on A) and followed
> the cluster start procedure on B,C,D,E (stop_app, reset, cluster all
> to B, start_app).

Reseting each of B, C, D should have been enough to inform A that they
have left the cluster, but getting the order of the commands even a bit
wrong might have led to your situation.

You were going from A, B, C, D to B, C, D, E.  If you wanted to destroy
the first cluster completely and create a new one reusing some of the
old machines, you should have "stop_app, reset, start_app" on each of
the original machines (maybe force_resetting the last one) and then run
the your commands to set up the new cluster.

If you just wanted to remove A and add E (i.e. keep the cluster but
change the machines in it), it should have been enough to "stop_app,
reset, start_app" on A and "stop_app, reset, cluster, start_app" on E.

> the cluster start procedure on B,C,D,E (stop_app, reset, cluster all
> to B, start_app).

If B is the only disc node, you should be able to break the cluster by
stopping all the other nodes and resetting B.

Hope this clears things up.

Cheers,
Alex

On Wed, Aug 03, 2011 at 04:20:57PM +0400, Eugene Kirpichov wrote:
> Hey.
> 
> I got the following error today: I was changing the composition of a
> 4-node RabbitMQ cluster (it was on nodes A,B,C,D and became B,C,D,E).
> I killed rabbitmq on nodes B,C,D (forgot to kill it on A) and followed
> the cluster start procedure on B,C,D,E (stop_app, reset, cluster all
> to B, start_app).
> 
> Is this expected behavior in such a scenario?
> Would the right way to avoid it be "kill on A before starting B,C,D,E cluster"?
> 
> =INFO REPORT==== 3-Aug-2011::04:50:23 ===
> Limiting to approx 924 file handles (829 sockets)
> 
> =ERROR REPORT==== 3-Aug-2011::04:50:23 ===
> Mnesia(rabbit at RACK2UNIT002): ** ERROR ** mnesia_event got
> {inconsistent_database, running_partitioned_network,
> rabbit at RACK2UNIT001}
> 
> =ERROR REPORT==== 3-Aug-2011::04:50:23 ===
> Mnesia(rabbit at RACK2UNIT002): ** ERROR ** mnesia_event got
> {inconsistent_database, starting_partitioned_network,
> rabbit at RACK2UNIT001}
> 
> =ERROR REPORT==== 3-Aug-2011::04:50:23 ===
> FAILED
> Reason: {error,
>             {unable_to_join_cluster,
>                 [rabbit at RACK2UNIT002,rabbit at RACK2UNIT001],
>                 {merge_schema_failed,
>                     "Bad cookie in table definition rabbit_queue:
> rabbit at RACK2UNIT002 =
> {cstruct,rabbit_queue,set,[rabbit at RACK2UNIT002],[],[],0,read_write,[],[],false,amqqueue,[name,durable,auto_delete,exclusive_owner,arguments,pid],[],[],{{1312,371791,515002},rabbit at RACK2UNIT001},{{3,1},{rabbit at RACK2UNIT002,{1312,372070,234006}}}},
> rabbit at RACK2UNIT001 =
> {cstruct,rabbit_queue,set,[rabbit at RACK2UNIT004,rabbit at RACK2UNIT003,rabbit at RACK2UNIT001],[],[],0,read_write,[],[],false,amqqueue,[name,durable,auto_delete,exclusive_owner,arguments,pid],[],[],{{1312,372061,609002},rabbit at RACK2UNIT001},{{3,2},{rabbit at RACK2UNIT003,{1312,372100,781007}}}}\n"}}}
> Stacktrace: [{rabbit_mnesia,init_db,3},
>              {rabbit_mnesia,init,0},
>              {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
>              {rabbit,run_boot_step,1},
>              {rabbit,'-start/2-lc$^0/1-0-',1},
>              {rabbit,start,2},
>              {application_master,start_it_old,4}]
> 
> =INFO REPORT==== 3-Aug-2011::04:50:24 ===
>     application: rabbit
>     exited: {bad_return,{{rabbit,start,[normal,[]]},
>                          {'EXIT',{rabbit,failure_during_boot}}}}
>     type: permanent
> 
> -- 
> Eugene Kirpichov
> Principal Engineer, Mirantis Inc. http://www.mirantis.com/
> Editor, http://fprog.ru/
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss