[rabbitmq-discuss] RabbitMQ 3.0 Policy ha-all delete issue

Simon MacMullen simon at rabbitmq.com
Tue Nov 20 17:20:09 GMT 2012


Hi. Thanks for the bug report.

So it seems like there's a race involving queues being deleted and 
having the message sent to them to start / stop mirroring. Hopefully 
this shouldn't be too big a deal to fix. I'll keep you informed.

Cheers, Simon

On 20/11/12 17:02, Mark Ward wrote:
> I should note that my testing clients are actively sending messages in 4
> static queues.  but for each message sent they are creating a dynamic queue
> and then later deleting the queue.  I think the issue could be happening
> wile the cluster is handling some of the more volatile queues that are
> created/deleted by the clients.
>
> What else I noticed.  I deleted the previous queue that was still tied to a
> policy that didn't exist.  Began running my test again with no policies.
> During the testing I created the policy to mirror the queues and received
> another error.  then after a few moments of I deleted the policy.  An error
> again.
>
> Now one of my more static queues was lost. It does not show up in the
> management anymore.   I think the queue "crashed"
>
> Some of my errors from the logs
>
> =INFO REPORT==== 20-Nov-2012::10:33:30 ===
> Adding mirror of queue 'dcf.action.status' in vhost 'MoBunnyMoProblems.DCF'
> on node 'rabbit at CUST1-MASTER': <2756.5031.0>
>
> =INFO REPORT==== 20-Nov-2012::10:33:30 ===
> Adding mirror of queue 'dcf.action.status' in vhost 'MoBunnyMoProblems.DCF'
> on node 'rabbit at RIOBARON-1': <2754.13177.4>
>
> =WARNING REPORT==== 20-Nov-2012::10:46:47 ===
> closing AMQP connection <0.32767.3> (192.168.10.104:37880 ->
> 192.168.10.57:5672):
> connection_closed_abruptly
>
> =ERROR REPORT==== 20-Nov-2012::10:49:28 ===
> ** Generic server <0.6.4> terminating
> ** Last message in was stop_mirroring
> ** When Server state == {q,
>                           {amqqueue,
>                            {resource,<<"MoBunnyMoProblems.DCF">>,queue,
>                             <<"dcf.action.status">>},
>                            true,false,none,
>                            [{<<"x-expires">>,signedint,100800000}],
>                            <0.6.4>,[],[],
>                            [{vhost,<<"MoBunnyMoProblems.DCF">>},
>                             {name,<<"ha-all">>},
>                             {pattern,<<"^dcf">>},
>                             {definition,[{<<"ha-mode">>,<<"all">>}]},
>                             {priority,0}],
>                            []},
>                           none,true,rabbit_variable_queue,
>                           {vqstate,
>                            {0,{[],[]}},
>                            {0,{[],[]}},
>                            {delta,undefined,0,undefined},
>                            {0,{[],[]}},
>                            {0,{[],[]}},
>                            10885,
>                            {0,nil},
>                            undefined,
>                            {0,nil},
>                            {qistate,
>
> "c:/rabbitmq.hold/db/rabbit at RIOOVERLORD-1-mnesia/queues/CTMLPLO20JY4AIHN3GGESPI7D",
>                             {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                               {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []}}},
>                              [{segment,0,
>
> "c:/rabbitmq.hold/db/rabbit at RIOOVERLORD-1-mnesia/queues/CTMLPLO20JY4AIHN3GGESPI7D/0.idx",
>                                {array,16384,0,undefined,
>                                 {{1000,1000,1000,1000,1000,1000,
>                                   {100,100,100,100,100,100,100,100,
>                                    {10,10,10,10,10,10,
>                                     {undefined,undefined,undefined,
>
> {{<<240,91,13,230,237,230,42,174,76,54,117,
>                                          111,236,199,4,71>>,
>                                        {message_properties,undefined,true,
>                                         false},
>                                        true},
>                                       del,ack},
>
> ..................... deleted bunch of stuff ......................
>
>                                     10,10},
>                                    100,100},
>
> 1000,1000,1000,1000,1000,1000,1000,1000,1000,
>                                   1000},
>                                  10000,10000,10000,10000,10000,10000,10000,
>                                  10000,10000}},
>                                0}]},
>                             #Ref<0.0.7.158837>,12066,262144,
>                             #Fun<rabbit_variable_queue.2.41635915>,
>                             {0,nil}},
>                            {{client_msstate,msg_store_persistent,
>
> <<158,88,38,57,48,220,55,115,72,72,33,241,89,104,
>                                106,124>>,
>                              {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                               {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []}}},
>                              {state,344140,
>
> "c:/rabbitmq.hold/db/rabbit at RIOOVERLORD-1-mnesia/msg_store_persistent"},
>                              rabbit_msg_store_ets_index,
>
> "c:/rabbitmq.hold/db/rabbit at RIOOVERLORD-1-mnesia/msg_store_persistent",
>                              <0.267.0>,348237,340043,352334,356431},
>                             {client_msstate,msg_store_transient,
>
> <<83,30,50,235,43,138,36,105,58,170,111,228,99,194,
>                                5,152>>,
>                              {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                               {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []}}},
>                              {state,323655,
>
> "c:/rabbitmq.hold/db/rabbit at RIOOVERLORD-1-mnesia/msg_store_transient"},
>                              rabbit_msg_store_ets_index,
>
> "c:/rabbitmq.hold/db/rabbit at RIOOVERLORD-1-mnesia/msg_store_transient",
>                              <0.262.0>,327752,319551,331849,335946}},
>
> true,0,#Fun<rabbit_amqqueue_process.5.77104289>,0,0,
>                            infinity,0,0,0,95,95,
>                            {rates,
>                             {{1353,430160,19630},155},
>                             {{1353,430160,19630},155},
>                             32.34796456653727,32.34796456653727,
>                             {1353,430165,27630}},
>                            {0,nil},
>                            {0,nil},
>                            {0,nil},
>                            {0,nil},
>                            95,95,
>                            {rates,
>                             {{1353,430160,19630},155},
>                             {{1353,430160,19630},155},
>                             32.34796456653727,32.34796456653727,
>                             {1353,430165,27630}}},
>                           {[{<2756.5629.0>,
>                              {consumer,<<"amq.ctag-ZHZP5Ybp_xZxsi5kvmuVfw">>,
>                               true}}],
>                            []},
>                           100800000,undefined,#Ref<0.0.8.36505>,
>                           #Ref<0.0.8.1338>,
>                           {state,fine,5000,#Ref<0.0.8.36524>},
>                           {0,nil},
>                           undefined,undefined,undefined,
>                           {dict,3,16,16,8,80,48,
>                            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                            {{[[<2756.5690.0>|#Ref<0.0.8.1812>],
>                               [<2754.15277.4>|#Ref<0.0.8.1819>]],
>                              [],[],[],[],[],[],[],
>                              [[<2754.15269.4>|#Ref<0.0.8.1825>]],
>                              [],[],[],[],[],[],[]}}},
>                           1,
>                           {{0,nil},{0,nil}},
>                           undefined,
>                           {dict,0,16,16,8,80,48,
>                            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                           undefined,undefined}
>
>
> ** Reason for termination ==
> ** {{badmatch,rabbit_mirror_queue_master},
>      [{rabbit_amqqueue_process,handle_call,3,[]},
>       {gen_server2,handle_msg,2,[]},
>       {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}
>
>
>
>
>
> --
> View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-3-0-Policy-ha-all-delete-issue-tp23527p23530.html
> Sent from the RabbitMQ mailing list archive at Nabble.com.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


-- 
Simon MacMullen
RabbitMQ, VMware


More information about the rabbitmq-discuss mailing list