Hi, Simone...<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>I am relatively new to Rabbitmq and would appreciate help in troubleshooting a recurring issue on a cluster, apologies for the long email.</div>
</blockquote><div><br></div><div>No problem! Details are usually helpful...</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>I run a 3 instances cluster in ec2, 2 disk nodes (A and B) 1 ram node (C), exchanges and queues are static and limited in number (less than 50), the volume of messages can reach a few thousands per second and the queues can occasionally grow up to a few hundred thousands until the processes manage to catch up, but this is well within our memory/disk high watermark. Rabbitmq is v. 2.8.4 on ubuntu 12.</div>
</blockquote><div><br></div><div>2.8.4 is now getting a bit old, and was from the middle of a sequence of bug fix releases, during which many things improved... you might want to consider upgrading. Before you do, see remarks below.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>I would like to better understand the crash report and perhaps have some ideas on what went wrong and how to more effectively troubleshoot issues (what more info should I collect before restarting the nodes, erlang processes list, mnesia tables, ets tables etc).</div>
<div><br></div><div><div>=CRASH REPORT==== 14-Feb-2013::10:36:54 ===</div><div> crasher:</div><div> initial call: rabbit_reader:init/4</div><div><span> pid: <</span><b>0.29283.387</b><span>></span></div>
<div> registered_name: []</div><div> exception error: bad argument</div><div> in function port_close/1</div><div> called as port_close(#Port<0.746540>)</div><div> in call from rabbit_net:maybe_fast_close/1</div>
<div> in call from rabbit_reader:start_connection/7</div><div> ancestors: [<0.29280.387>,rabbit_tcp_client_sup,rabbit_sup,<0.161.0>]</div><div> messages: []</div><div> links: [<0.29280.387>]</div>
<div> dictionary: [{{channel,10},</div><div> {<0.29364.387>,{method,rabbit_framing_amqp_0_9_1}}},</div><div> {{ch_pid,<0.29338.387>},{7,#Ref<0.0.2158.60093>}},</div>
<div> {{ch_pid,<0.29333.387>},{6,#Ref<0.0.2158.60085>}},</div><div> {{ch_pid,<0.29325.387>},{5,#Ref<0.0.2158.60053>}},</div><div> {{channel,3},</div>
<div> {<0.29313.387>,{method,rabbit_framing_amqp_0_9_1}}},</div><div> {{ch_pid,<0.29305.387>},{2,#Ref<0.0.2158.60002>}},</div><div> {{channel,4},</div>
<div> {<0.29321.387>,{method,rabbit_framing_amqp_0_9_1}}},</div><div> {{channel,11},</div><div> {<0.29370.387>,{method,rabbit_framing_amqp_0_9_1}}},</div><div>
{{ch_pid,<0.29313.387>},{3,#Ref<0.0.2158.60017>}},</div><div> {{ch_pid,<0.29299.387>},{1,#Ref<0.0.2158.59976>}},</div><div> {{ch_pid,<0.29346.387>},{8,#Ref<0.0.2158.60112>}},</div>
<div> {{ch_pid,<0.29370.387>},{11,#Ref<0.0.2158.60189>}},</div><div> {{channel,7},</div><div> {<0.29338.387>,{method,rabbit_framing_amqp_0_9_1}}},</div>
<div> {{channel,9},</div><div> {<0.29356.387>,{method,rabbit_framing_amqp_0_9_1}}},</div><div> {{ch_pid,<0.29321.387>},{4,#Ref<0.0.2158.60034>}},</div>
<div> {{ch_pid,<0.29364.387>},{10,#Ref<0.0.2158.60166>}},</div><div> {{ch_pid,<0.29356.387>},{9,#Ref<0.0.2158.60140>}},</div><div> {{channel,8},</div>
<div> {<0.29346.387>,{method,rabbit_framing_amqp_0_9_1}}},</div><div> {{channel,5},</div><div> {<0.29325.387>,{method,rabbit_framing_amqp_0_9_1}}},</div><div>
{{channel,1},</div><div> {<0.29299.387>,</div><div> {content_body,</div><div> {'basic.publish',0,<<"some_exchange">>,<<>>,false,</div>
<div> false},</div><div> 1048189,</div><div> {content,60,none,</div><div> <<BYTES IN HERE>>, --> this showed which process was sending the message</div>
<div> rabbit_framing_amqp_0_9_1,</div><div> [<<MORE BYTES IN HERE>>] --> This I haven't been able to decode, it is fairly big, is it truncated?</div>
</div></blockquote><div><br></div><div>Unfortunately, nothing springs to mind immediately to pursue from this...</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
</div><div>And in the logs we can find the pid <b>0.29283.387 </b>right before the crash:</div><div><br></div><div>=INFO REPORT==== 14-Feb-2013::10:31:46 ===</div><div><span>accepting AMQP connection <<b>0.29283.387</b>> (10.xx.xx.xx:58622 -> 10.</span>xx.xx.xx<span>:5672)</span></div>
<div><br></div><div>=INFO REPORT==== 14-Feb-2013::10:31:46 ===</div><div><span>accepting AMQP connection <0.29287.387> (10.</span>xx.xx.xx<span>:58623 -> 10.</span>xx.xx.xx<span>:5672)</span></div><div><br></div>
<div>=WARNING REPORT==== 14-Feb-2013::10:32:27 ===</div><div><span>closing AMQP connection <0.27107.387> (10.</span>xx.xx.xx<span>:50882 -> 10.</span>xx.xx.xx<span>:5672):</span></div><div>connection_closed_abruptly</div>
</blockquote><div><br></div><div>This could be pretty much anything, from client misbehavior to connection disruption...</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Looking at the rabbitmqctl report I have not been able to map the memory consumption to something specific yet.<br></div></blockquote><div><br></div><div>I'd proceed with an upgrade to a more recent Rabbit... check here first:</div>
<div><br></div><div><a href="http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/">http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/</a></div><div><br></div><div>And if none of the changes in 3.0.x are going to provide you short term inconvenience, then try going straight to the latest 3.0.2; if there are 3.0 changes that you think will bother you or require changes to your apps or infrastructure, then jump to 2.8.6 for now... it was the seventh and last of the 2.8.x series and contains a pile of incremental fixes that may help with this (otherwise tricky to diagnose from what we have available right now) problem.</div>
<div><br></div><div>Best regards,</div><div>Jerry</div><div><br></div></div>