<div dir="ltr">BTW, here are the sasl logs from another node in the cluster:<div><br></div><div><div>=CRASH REPORT==== 13-Feb-2014::05:14:36 ===</div><div> crasher:</div><div> initial call: gen:init_it/6</div><div> pid: <0.987.0></div>
<div> registered_name: []</div><div> exception exit: {{badmatch,{error,not_found}},</div><div> [{rabbit_amqqueue_process,i,2,[]},</div><div> {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},</div>
<div> {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},</div><div> {rabbit_amqqueue_process,emit_stats,2,[]},</div><div> {rabbit_event,if_enabled,3,[]},</div>
<div> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',</div><div> 6,[]},</div><div> {rabbit_amqqueue_process,terminate_shutdown,2,[]},</div>
<div> {gen_server2,terminate,3,[]}]}</div><div> in function gen_server2:terminate/3</div><div> ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.782.0>]</div><div> messages: [{'$gen_cast',</div>
<div> {run_backing_queue,rabbit_variable_queue,</div><div> #Fun<rabbit_variable_queue.26.70600163>}},</div><div> {'EXIT',<0.988.0>,normal}]</div>
<div> links: [<0.954.0>]</div><div> dictionary: [{{credit_from,<0.944.0>},1671},</div><div> {{credit_to,<0.24877.6355>},2},</div><div> {credit_blocked,[]},</div><div>
{delegate,delegate_0},</div><div> {fhc_age_tree,{0,nil}},</div><div> {guid,{{<a href="tel:2283490857" value="+12283490857" target="_blank">2283490857</a>,778293189,3964001052,3912480778},1}}]</div>
<div> trap_exit: true</div><div> status: running</div>
<div> heap_size: 6772</div><div> stack_size: 27</div><div> reductions: 28827118159</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 13-Feb-2014::05:14:36 ===</div><div> Supervisor: {local,</div>
<div> rabbit_mirror_queue_slave_sup}</div><div> Context: child_terminated</div><div> Reason: {{badmatch,{error,not_found}},</div><div> [{rabbit_amqqueue_process,i,2,[]},</div>
<div> {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},</div><div> {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},</div><div> {rabbit_amqqueue_process,emit_stats,2,[]},</div>
<div> {rabbit_event,if_enabled,3,[]},</div><div> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},</div><div> {rabbit_amqqueue_process,terminate_shutdown,2,[]},</div>
<div> {gen_server2,terminate,3,[]}]}</div><div> Offender: [{pid,<0.987.0>},</div><div> {name,rabbit_mirror_queue_slave},</div><div> {mfargs,{rabbit_mirror_queue_slave,start_link,undefined}},</div>
<div> {restart_type,temporary},</div><div> {shutdown,4294967295},</div><div> {child_type,worker}]</div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Feb 14, 2014 at 9:43 AM, Jason McIntosh <span dir="ltr"><<a href="mailto:mcintoshj@gmail.com" target="_blank">mcintoshj@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><br></div>RabbitMQ 3.2.0<div>Erlang R16B02-1</div><div><br></div><div>We have a queue that basically stopped doing anything intelligent. Here are the results. What's bad about this - it appears that messages continued to publish and didn't hit the dead letter exchange - they just disappeared. In this architecture, we've got a fanout exchange that publishes to two queues. One of the queues is working fine still. Our second queue though is what dropped off. Publishing though hasn't failed so I'm worried we've lost data for the last data. Any input would be welcome on this. Here's the second queues information from the management gui:<br>
</div><div><table style="border-collapse:collapse;border-width:1px;border-bottom-style:solid;border-bottom-color:rgb(204,204,204);margin-bottom:0px;font-family:Verdana,sans-serif"><tbody><tr><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)))">
<small style="font-size:0.8em;color:rgb(136,136,136)">cluster@</small>rabbitmqm10p</td><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:center!important">
</td><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:center!important">
<acronym title="x-dead-letter-exchange: vinclassifier.errors" style="background-color:rgb(170,221,221);padding:2px 4px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;border:none">DLX</acronym> <acronym title="x-dead-letter-routing-key: vinclassifier.retail.listings.errors" style="background-color:rgb(170,221,221);padding:2px 4px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;border:none">DLK</acronym> <acronym title="durable: true" style="background-color:rgb(170,221,221);padding:2px 4px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;border:none">D</acronym> <acronym title="x-ha-policy: all" style="background-color:rgb(170,221,221);padding:2px 4px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;border:none">Args</acronym></td>
<td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:center!important">
</td><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:center!important">
Active</td><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:right!important">
<span style="color:rgb(136,136,136)">?</span></td><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:right!important">
<span style="color:rgb(136,136,136)">?</span></td><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:right!important">
<span style="color:rgb(136,136,136)">?</span></td><td style="font-size:12px;line-height:17px;font-family:Verdana,sans-serif;padding:4px;vertical-align:top;min-width:5em;width:auto;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);border-right-width:1px;border-right-style:solid;border-right-color:rgb(204,204,204);background-image:-webkit-gradient(linear,0% 0%,0% 100%,from(rgb(248,248,248)),to(rgb(255,255,255)));text-align:right!important">
0.00/s</td></tr></tbody></table></div><div><br></div><div>When I try and select the queue, I just get an error message:</div><div><span style="color:rgb(72,72,72);font-family:Verdana,sans-serif;font-size:12px">TypeError: Cannot read property 'ram_msg_count' of undefined</span><br>
</div><div><br></div><div>Any help/advice here? Is there some way I can change this queue so I do NOT lose messages and publishes fail?? I thought publisher confirms (need to verify they're on) would have taken care of this situation - that the message would have had to have been consumed or persisted to disk for all queues or publishing would have been rejected.</div>
<div>Jason</div><div><br></div><div><br></div><div><div><br></div><div>=CRASH REPORT==== 13-Feb-2014::05:14:36 ===</div><div> crasher:</div><div> initial call: gen:init_it/6</div><div> pid: <0.367.0></div><div>
registered_name: []</div><div> exception exit: {{badmatch,{error,not_found}},</div><div> [{rabbit_mirror_queue_master,stop_all_slaves,2,[]},</div><div> {rabbit_mirror_queue_master,delete_and_terminate,2,[]},</div>
<div> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',</div><div> 6,[]},</div><div> {rabbit_amqqueue_process,terminate_shutdown,2,[]},</div>
<div> {gen_server2,terminate,3,[]},</div><div> {proc_lib,wake_up,3,</div><div> [{file,"proc_lib.erl"},{line,249}]}]}</div><div> in function gen_server2:terminate/3</div>
<div> ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.154.0>]</div><div> messages: []</div><div> links: [<0.250.0>,#Port<0.17147>]</div><div> dictionary: [{{ch,<17654.9226.6150>},</div><div>
{cr,<17654.9226.6150>,#Ref<0.0.18055.20563>,</div><div> {[],[26925191]},</div><div> 1,</div><div> {queue,</div><div> [{<17654.9226.6150>,</div>
<div> {consumer,<<"amq.ctag-LPmzPvp2doZ9pYs-cEEcFg">>,</div><div> true,[]}}],</div><div> [],1},</div><div> {qstate,<17654.21979.6150>,suspended,{0,nil}},</div>
<div> 4}},</div><div> {credit_blocked,[]},</div><div> {{ch,<17659.4312.6334>},</div><div> {cr,<17659.4312.6334>,#Ref<0.0.18273.227308>,</div>
<div> {[],[26925208]},</div><div> 1,</div><div> {queue,</div><div> [{<17659.4312.6334>,</div><div> {consumer,<<"amq.ctag--3Kwc_Q-QS9kcpZ9U--8-Q">>,</div>
<div> true,[]}}],</div><div> [],1},</div><div> {qstate,<17659.2894.6334>,suspended,{0,nil}},</div><div> 19}},</div>
<div> {{ch,<17659.3911.6334>},</div><div> {cr,<17659.3911.6334>,#Ref<0.0.18273.227286>,</div><div> {[26925232,26925226],[26925214]},</div><div> 1,</div>
<div> {queue,[],[],0},</div><div> {qstate,<17659.2051.6334>,active,{0,nil}},</div><div> 22}},</div><div> {{#Ref<0.0.0.36427>,fhc_handle},</div>
<div> {handle,</div><div> {file_descriptor,prim_file,{#Port<0.17147>,132}},</div><div> 118224,false,5136,infinity,</div><div> [[<<192,0,0,0,1,154,216,155>>],</div>
<div> [<<192,0,0,0,1,154,216,151>>],</div><div> [<<192,0,0,0,1,154,216,150>>],</div><div> [<<192,0,0,0,1,154,216,149>>],</div>
<div> [<<192,0,0,0,1,154,216,148>>],</div><div> [<<192,0,0,0,1,154,216,147>>],</div><div> [<<192,0,0,0,1,154,216,146>>],</div>
<div> [<<192,0,0,0,1,154,216,144>>],</div><div> [<<192,0,0,0,1,154,216,142>>],</div><div> [<<192,0,0,0,1,154,216,143>>],</div>
<div> [<<192,0,0,0,1,154,216,141>>],</div><div> [<<192,0,0,0,1,154,216,140>>],</div><div>.,...</div><div><br></div><div><br></div><div><div><br></div>
<div>
=SUPERVISOR REPORT==== 13-Feb-2014::05:14:36 ===</div><div> Supervisor: {local,rabbit_amqqueue_sup}</div><div> Context: child_terminated</div><div> Reason: {{badmatch,{error,not_found}},</div><div> [{rabbit_mirror_queue_master,stop_all_slaves,2,[]},</div>
<div> {rabbit_mirror_queue_master,delete_and_terminate,2,[]},</div><div> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},</div><div> {rabbit_amqqueue_process,terminate_shutdown,2,[]},</div>
<div> {gen_server2,terminate,3,[]},</div><div> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</div><div> Offender: [{pid,<0.367.0>},</div><div> {name,rabbit_amqqueue},</div>
<div> {mfargs,{rabbit_amqqueue_process,start_link,undefined}},</div><div> {restart_type,temporary},</div><div> {shutdown,4294967295},</div><div> {child_type,worker}]</div>
<div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 13-Feb-2014::10:59:28 ===</div><div> Supervisor: {<0.19778.5266>,</div><div> amqp_channel_sup_sup}</div><div> Context: shutdown_error</div>
<div> Reason: shutdown</div><div> Offender: [{nb_children,1},</div><div> {name,channel_sup},</div><div> {mfargs,</div><div> {amqp_channel_sup,start_link,[direct,<0.20460.5266>]}},</div>
<div> {restart_type,temporary},</div><div> {shutdown,brutal_kill},</div><div> {child_type,supervisor}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 13-Feb-2014::11:02:34 ===</div>
<div> Supervisor: {<0.852.5267>,amqp_channel_sup_sup}</div><div> Context: shutdown_error</div><div> Reason: shutdown</div><div> Offender: [{nb_children,1},</div><div> {name,channel_sup},</div>
<div> {mfargs,</div><div> {amqp_channel_sup,start_link,[direct,<0.2623.5267>]}},</div><div> {restart_type,temporary},</div><div> {shutdown,brutal_kill},</div>
<div> {child_type,supervisor}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 13-Feb-2014::11:03:24 ===</div><div> Supervisor: {<0.4628.5267>,amqp_channel_sup_sup}</div><div> Context: shutdown_error</div>
<div> Reason: shutdown</div><div> Offender: [{nb_children,1},</div><div> {name,channel_sup},</div><div> {mfargs,</div><div> {amqp_channel_sup,start_link,[direct,<0.5878.5267>]}},</div>
<div> {restart_type,temporary},</div><div> {shutdown,brutal_kill},</div><div> {child_type,supervisor}]</div></div><div><br></div><div><div><br></div><div>=CRASH REPORT==== 13-Feb-2014::11:12:31 ===</div>
<div> crasher:</div><div> initial call: gen:init_it/6</div><div> pid: <0.4699.5268></div><div> registered_name: []</div><div> exception exit: {{badmatch,true},</div><div> [{rabbit_queue_index,init,2,[]},</div>
<div> {rabbit_variable_queue,init,5,[]},</div><div> {rabbit_mirror_queue_master,init,3,[]},</div><div> {rabbit_amqqueue_process,declare,3,[]},</div><div> {gen_server2,handle_msg,2,[]},</div>
<div> {proc_lib,init_p_do_apply,3,</div><div> [{file,"proc_lib.erl"},{line,239}]}]}</div><div> in function gen_server2:terminate/3</div><div> ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.154.0>]</div>
<div> messages: []</div><div> links: [<0.250.0>]</div><div> dictionary: [{{xtype_to_module,direct},rabbit_exchange_type_direct}]</div><div> trap_exit: true</div><div> status: running</div><div> heap_size: 1598</div>
<div> stack_size: 27</div><div> reductions: 1156</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 13-Feb-2014::11:12:31 ===</div><div> Supervisor: {local,rabbit_amqqueue_sup}</div><div> Context: child_terminated</div>
<div> Reason: {{badmatch,true},</div><div> [{rabbit_queue_index,init,2,[]},</div><div> {rabbit_variable_queue,init,5,[]},</div><div> {rabbit_mirror_queue_master,init,3,[]},</div>
<div> {rabbit_amqqueue_process,declare,3,[]},</div><div> {gen_server2,handle_msg,2,[]},</div><div> {proc_lib,init_p_do_apply,3,</div><div> [{file,"proc_lib.erl"},{line,239}]}]}</div>
<div> Offender: [{pid,<0.4699.5268>},</div><div> {name,rabbit_amqqueue},</div><div> {mfargs,{rabbit_amqqueue_process,start_link,undefined}},</div><div> {restart_type,temporary},</div>
<div> {shutdown,4294967295},</div><div> {child_type,worker}]</div></div><div><br></div><div><div>=SUPERVISOR REPORT==== 13-Feb-2014::11:35:08 ===</div><div> Supervisor: {<0.6708.5271>,amqp_channel_sup_sup}</div>
<div> Context: shutdown_error</div><div> Reason: shutdown</div><div> Offender: [{nb_children,1},</div><div> {name,channel_sup},</div><div> {mfargs,</div><div> {amqp_channel_sup,start_link,[direct,<0.7855.5271>]}},</div>
<div> {restart_type,temporary},</div><div> {shutdown,brutal_kill},</div><div> {child_type,supervisor}]</div></div><span class="HOEnZb"><font color="#888888"><div><br></div>
<div><br></div>-- <br><div dir="ltr">Jason McIntosh<br>
<a href="https://github.com/jasonmcintosh/" target="_blank">https://github.com/jasonmcintosh/</a><br><a href="tel:573-424-7612" value="+15734247612" target="_blank">573-424-7612</a></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Jason McIntosh<br><a href="https://github.com/jasonmcintosh/" target="_blank">https://github.com/jasonmcintosh/</a><br>573-424-7612</div>
</div>