<div dir="ltr">One of our clusters had a problem over the weekend. Ops were upgrading the Load Balancer and restatrted the RabbitMQ service on both nodes at 15:00. Everything seemed to come back up Ok but the following errors started showing later in the day at 17:15<div>
<br></div><div>I have copied some of the SASL log below. Any ideas why this would have started happening?</div><div><br></div><div><br></div><div><div>=CRASH REPORT==== 5-Apr-2014::19:14:34 ===</div><div> crasher:</div><div>
initial call: gen:init_it/6</div><div> pid: <0.9454.1></div><div> registered_name: []</div><div> exception exit: {function_clause,</div><div> [{rabbit_channel,handle_info,</div><div>
[{{#Ref<0.0.1.206248>,rabbit@NWPAPPRMQA01},</div><div> [{ok,<8069.506.0>,{ok,0,0}}]},</div><div> {ch,running,rabbit_framing_amqp_0_9_1,1,<0.9451.1>,</div>
<div> <0.9451.1>,<0.9445.1>,</div><div> <<"<rabbit@NWPAPPRMQA02.1.9445.1>">>,</div><div> {lstate,<0.9453.1>,false,false},</div>
<div> none,1,</div><div> {[],[]},</div><div> {user,<<"guest">>,</div><div> [administrator],</div><div>
rabbit_auth_backend_internal,</div><div> {internal_user,<<"guest">>,</div><div> <<54,19,230,202,176,82,197,60,61,40,94,249,70,83,81,</div>
<div> 243,160,53,79,216>>,</div><div> [administrator]}},</div><div> <<"/">>,<<"<b>aliveness-test</b>">>,.....</div>
</div><div><br></div><div><div>=SUPERVISOR REPORT==== 5-Apr-2014::19:14:34 ===</div><div> Supervisor: {<0.9446.1>,amqp_channel_sup_sup}</div><div> Context: child_terminated</div><div> Reason: {function_clause,</div>
<div> [{rabbit_channel,handle_info,</div><div> [{{#Ref<0.0.1.206248>,rabbit@NWPAPPRMQA01},</div><div> [{ok,<8069.506.0>,{ok,0,0}}]},</div>
<div> {ch,running,rabbit_framing_amqp_0_9_1,1,<0.9451.1>,</div><div> <0.9451.1>,<0.9445.1>,</div><div> <<"<rabbit@NWPAPPRMQA02.1.9445.1>">>,</div>
<div> {lstate,<0.9453.1>,false,false},</div><div> none,1,</div><div> {[],[]},</div><div> {user,<<"guest">>,</div>
<div> [administrator],</div><div> rabbit_auth_backend_internal,</div><div> {internal_user,<<"guest">>,</div>
<div> <<54,19,230,202,176,82,197,60,61,40,94,</div><div> 249,70,83,81,243,160,53,79,216>>,</div><div> [administrator]}},</div>
<div> <<"/">>,<<"aliveness-test">>,</div></div><div><br></div><div><div>=SUPERVISOR REPORT==== 5-Apr-2014::19:14:34 ===</div><div> Supervisor: {<0.9452.1>,rabbit_channel_sup}</div>
<div> Context: shutdown</div><div> Reason: reached_max_restart_intensity</div><div> Offender: [{pid,<0.9454.1>},</div><div> {name,channel},</div><div> {mfargs,</div>
<div> {rabbit_channel,start_link,</div><div> [1,<0.9451.1>,<0.9451.1>,<0.9445.1>,</div><div> <<"<rabbit@NWPAPPRMQA02.1.9445.1>">>,</div>
<div> rabbit_framing_amqp_0_9_1,</div><div> {user,<<"guest">>,</div><div> [administrator],</div><div> rabbit_auth_backend_internal,</div>
<div> {internal_user,<<"guest">>,</div><div> <<54,19,230,202,176,82,197,60,61,40,94,249,</div><div> 70,83,81,243,160,53,79,216>>,</div>
<div> [administrator]}},</div><div> <<"/">>,</div><div> [{<<"publisher_confirms">>,bool,true},</div>
<div> {<<"exchange_exchange_bindings">>,bool,true},</div><div> {<<"basic.nack">>,bool,true},</div><div> {<<"consumer_cancel_notify">>,bool,true},</div>
<div> {<<"connection.blocked">>,bool,true},</div><div> {<<"authentication_failure_close">>,bool,true}],</div><div> <0.9448.1>,<0.9453.1>]}},</div>
<div> {restart_type,intrinsic},</div><div> {shutdown,4294967295},</div><div> {child_type,worker}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 5-Apr-2014::19:14:34 ===</div>
<div> Supervisor: {<0.362.0>,mirrored_supervisor}</div><div> Context: child_terminated</div><div> Reason: killed</div><div> Offender: [{pid,<0.9479.1>},</div><div> {name,rabbit_mgmt_db},</div>
<div> {mfargs,{rabbit_mgmt_db,start_link,[]}},</div><div> {restart_type,permanent},</div><div> {shutdown,4294967295},</div><div> {child_type,worker}]</div>
<div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 5-Apr-2014::19:14:35 ===</div><div> Supervisor: {<0.362.0>,mirrored_supervisor}</div><div> Context: start_error</div><div> Reason: {already_started,<8069.464.0>}</div>
<div> Offender: [{pid,<0.9479.1>},</div><div> {name,rabbit_mgmt_db},</div><div> {mfargs,{rabbit_mgmt_db,start_link,[]}},</div><div> {restart_type,permanent},</div>
<div> {shutdown,4294967295},</div><div> {child_type,worker}]</div><div><br></div><div>Thanks</div><div><br></div><div><br></div>-- <br>Patrick Long - Munkiisoft Ltd
</div></div>