[rabbitmq-discuss] Unable to join cluster: "Bad cookie in table definition rabbit_queue"

[Eugene Kirpichov]

Hey.

I got the following error today: I was changing the composition of a
4-node RabbitMQ cluster (it was on nodes A,B,C,D and became B,C,D,E).
I killed rabbitmq on nodes B,C,D (forgot to kill it on A) and followed
the cluster start procedure on B,C,D,E (stop_app, reset, cluster all
to B, start_app).

Is this expected behavior in such a scenario?
Would the right way to avoid it be "kill on A before starting B,C,D,E cluster"?

=INFO REPORT==== 3-Aug-2011::04:50:23 ===
Limiting to approx 924 file handles (829 sockets)

=ERROR REPORT==== 3-Aug-2011::04:50:23 ===
Mnesia(rabbit at RACK2UNIT002): ** ERROR ** mnesia_event got
{inconsistent_database, running_partitioned_network,
rabbit at RACK2UNIT001}

=ERROR REPORT==== 3-Aug-2011::04:50:23 ===
Mnesia(rabbit at RACK2UNIT002): ** ERROR ** mnesia_event got
{inconsistent_database, starting_partitioned_network,
rabbit at RACK2UNIT001}

=ERROR REPORT==== 3-Aug-2011::04:50:23 ===
FAILED
Reason: {error,
            {unable_to_join_cluster,
                [rabbit at RACK2UNIT002,rabbit at RACK2UNIT001],
                {merge_schema_failed,
                    "Bad cookie in table definition rabbit_queue:
rabbit at RACK2UNIT002 =
{cstruct,rabbit_queue,set,[rabbit at RACK2UNIT002],[],[],0,read_write,[],[],false,amqqueue,[name,durable,auto_delete,exclusive_owner,arguments,pid],[],[],{{1312,371791,515002},rabbit at RACK2UNIT001},{{3,1},{rabbit at RACK2UNIT002,{1312,372070,234006}}}},
rabbit at RACK2UNIT001 =
{cstruct,rabbit_queue,set,[rabbit at RACK2UNIT004,rabbit at RACK2UNIT003,rabbit at RACK2UNIT001],[],[],0,read_write,[],[],false,amqqueue,[name,durable,auto_delete,exclusive_owner,arguments,pid],[],[],{{1312,372061,609002},rabbit at RACK2UNIT001},{{3,2},{rabbit at RACK2UNIT003,{1312,372100,781007}}}}\n"}}}
Stacktrace: [{rabbit_mnesia,init_db,3},
             {rabbit_mnesia,init,0},
             {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
             {rabbit,run_boot_step,1},
             {rabbit,'-start/2-lc$^0/1-0-',1},
             {rabbit,start,2},
             {application_master,start_it_old,4}]

=INFO REPORT==== 3-Aug-2011::04:50:24 ===
    application: rabbit
    exited: {bad_return,{{rabbit,start,[normal,[]]},
                         {'EXIT',{rabbit,failure_during_boot}}}}
    type: permanent

-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/