[rabbitmq-discuss] Queue Deleted and Permissions Corrupted While Testing Flow Control

patrickjnewell at gmail.com patrickjnewell at gmail.com
Mon Apr 14 14:12:32 BST 2014


Hello,

While testing disk-based flow control, the queue my test client was 
consuming from suddenly got deleted and the user permissions appeared to 
get corrupted.  I was running the broker on a CentOS 6.5 VM and had 
recently upgraded the broker from 3.2.1 to 3.3.0.  The broker had 32 GB of 
available disk space with a 48 MB low watermark.  The performed the test as 
follows:

1) Start a producer that sends 4 million persistent, 10 kb messages to a 
durable queue
2) Wait for the producer to block due to disk-based flow control (right 
around 3.1 million messages)
3) Start a consumer to drain the queue (prefetch count of 8, acknowledges 
one message at a time), allowing the producer to continue to send messages

About 1 minute into step 3, the producer and consumer stopped suddenly and 
were disconnected about 1 minute later (the consumer received a Cancel). 
 The queue disappeared from the management dashboard around the same time. 
 About 30 seconds later, I was unable to use the dashboard at all -- my 
credentials appeared to be invalid and I was seeing 500 errors in the 
browser.

The 500 error was as follows:
{error,badarg,
       [{gen_event,send,2},
        {rabbit_access_control,check_user_login,2},
        {rabbit_mgmt_util,is_authorized,6},
        {webmachine_resource,resource_call,3},
        {webmachine_resource,do,3}
        {webmachine_decision_core,resource_call,1},
        {webmachine_decision_core,decision,1},
        {webmachine_decision_core,handle_request,2}]}

I ran rabbitmqctl and saw that the users and virtual hosts I had created 
still existed.  However, all exchanges and queues were no longer listed. 
 Additionally, rabbitmqctl appeared to crash whenever I ran 
"list_user_permissions" or "list_permissions":
[root at pnewellrabbitmqvm ~]# rabbitmqctl list_user_permissions admin
Listing permissions for user "admin" ...
Error: {noproc,{gen_server2,call,
                            
[worker_pool,{next_free,<4967.1308.0>},infinity]}}
[root at pnewellrabbitmqvm ~]# rabbitmqctl list_permissions
Listing permissions in vhost "/" ...
Error: {noproc,{gen_server2,call,
                            
[worker_pool,{next_free,<4967.1317.0>},infinity]}}

The sasl.log file listed several crash reports.  It looks like the 
persistent message store process(?) crashed when it was reading a message 
from disk, which caused subsequent crashes.  I included the log output 
below.

After resetting the broker (rabbitmqctl stop_app, reset, start_app) and 
recreating the objects, I ran the same test again and reproduced the same 
issue.  This time, however, the producer had managed to finish sending its 
messages and the consumer had received about 1.3 million messages before 
the queue and permissions crashed.  I re-ran the same test 3 or 4 more 
times, but did not reproduce the issue.

Any idea what might have caused this?  My best guess is that something 
might have gotten corrupted when I originally performed the upgrade -- I 
stopped the service (/sbin/service rabbitmq-server stop), upgraded the 
broker and restarted it.  The broker was empty with no messages persisted 
at the time of the upgrade.

Any help would be much appreciated.

Thanks,
-Pat


Here's the cleaned up sasl-log output (if I took out anything important, 
please let me know):
=CRASH REPORT==== 9-Apr-2014::18:42:05 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.214.0>
    registered_name: []
    exception exit: {{badmatch,
                      {error,
                       {misread,
                        [{old_state,
                          {client_msstate,msg_store_persistent,
                          
 <<184,251,38,246,192,82,244,149,210,80,115,109,162,
                             244,176,61>>,
                           {dict,1,16,16,8,80,48,
                            {[],[],[],[],'...'},
                            {{'...'}}},
                           {state,233544,"/data..."},
                           rabbit_msg_store_ets_index,"/data/rabbit...",
                           <0.210.0>,237641,229447,241738,245835}},
                         {file_num,681},
                         {offset,16669770},
                         {msg_id,
                          
<<160,82,77,79,243,175,93,42,1,219,170,168,159,121,
                            222,119>>},
                         {read,
                          {ok,
                          
 <<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1,
                             219,170,168,159,121,222,119,131,104,6,100,0,13,
                             98,97,115,105,99,95,109,101,115,115,97,103,101,
                             104,4,100,0,8,114,101,115,111,117,114,99,101,
                             109,0,0,0,2,100,49,100,0,8,101,120,99,104,97,
                            
 110,103,101,109,0,0,0,19,66,105,108,108,105,110,
                             103,78,111,116,105,102,105,99,97,116,105,111,
                             110,108,0,0,0,1,109,0,0,0,10,97,112,112,49,46,
                            
 100,101,98,117,103,106,104,6,100,0,7,99,111,110,
                             116,101,110,116,97,60,100,0,4,110,111,110,101,
                             109,0,0,0,29,152,0,24,97,112,112,108,105,99,97,
                             116,105,111,110,47,111,99,116,101,116,45,115,
                             116,114,101,97,109,2,0,100,0,25,114,97,98,98,
                            
 105,116,95,102,114,97,109,105,110,103,95,97,109,
                             113,112,95,48,95,57,95,49,108,0,0,0,1,109,0,0,
                            
 40,183,123,34,117,117,105,100,34,58,34,52,48,53,
                             97,98,51,48,97,45,53,57,55,98,45,52,102,97,57,
                             45,57,101,100,100,45,50,102,100,97,51,56,57,50,
                             57,100,51,50,34,44,34,115,101,113,117,101,110,
                            
 99,101,34,58,49,56,55,53,54,56,44,34,97,112,112,
                             108,105,99,97,116,105,111,110,34,58,34,97,112,
                            
 112,48,34,44,34,101,118,101,110,116,34,58,34,69,
                            
 82,82,79,82,34,44,34,100,97,116,97,34,58,123,34,
                            
 115,116,114,105,110,103,34,58,34,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            
 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                             97,97,97,97,46,46,46>>}},
                         {proc_dict,
                          [{random_seed,{1397,21447,511}},
                           {{"...",fhc_file},{file,1,true}},
                           {'$ancestors',
                            [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
                           {process_name,{rabbit_amqqueue_process,{'...'}}},
                           {fhc_age_tree,{2,{'...'}}},
                           {{ch,<0.901.0>},
                            {cr,<0.901.0>,#Ref<0.0.0.27418>,'...'}},
                           {credit_blocked,[]},
                           {{#Ref<0.0.0.1152>,fhc_handle},{handle,'...'}},
                           {{"...",'...'},{'...'}},
                           {{'...'},{'...'}},
                           {{'...'},50},
                           {'$initial_call',{'...'}},
                           {guid,{'...'}},
                           {{'...'},1350}]}]}}},
                     [{rabbit_msg_store,read_from_disk,2},
                      {rabbit_msg_store,client_read3,3},
                      {rabbit_msg_store,safe_ets_update_counter,5},
                      {rabbit_variable_queue,with_msg_store_state,3},
                      {rabbit_variable_queue,read_msg,3},
                      {rabbit_variable_queue,fetch,2},
                      {rabbit_mirror_queue_master,fetch,2},
                      {rabbit_amqqueue_process,fetch,2}]}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
    messages: [{'$gen_cast',
                   {deliver,
                    {delivery,false,false,<0.552.0>,
                     {basic_message,
                      
{resource,<<"d1">>,exchange,<<"BillingNotification">>},
                      [<<"app1.debug">>],
                      {content,60,
                       {'P_basic',<<"application/octet-stream">>,undefined,
                        
undefined,2,0,undefined,undefined,undefined,undefined,
                        undefined,undefined,undefined,undefined,undefined},
                      
 <<152,0,24,97,112,112,108,105,99,97,116,105,111,110,47,
                         111,99,116,101,116,45,115,116,114,101,97,109,2,0>>,
                       rabbit_framing_amqp_0_9_1,
                      
 [<<"{\"uuid\":\"dae09ba7-bf47-455c-b512-49d434fb7c3f\",\"sequence\":2363388,\"application\":\"app2\",\"event\":\"DEBUG\",\"data\":{\"string\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...">>]},
                      
<<30,161,133,41,140,102,238,100,34,46,141,56,48,162,217,
                        14>>,
                      true},
                     undefined},
                    false,flow}},

                    // Lots of queued up messages, acks, etc...

                  '...']
    links: [<0.211.0>]
    dictionary: [{random_seed,{1397,21447,511}},
                  {process_name,
                      {rabbit_amqqueue_process,
                          {resource,<<"d1">>,queue,<<"Appconnect">>}}},
                  {fhc_age_tree,{0,nil}},
                  {{ch,<0.901.0>},
                   {cr,<0.901.0>,#Ref<0.0.0.27418>,
                       {[{187567,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                         {187566,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}],
                        [{187561,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                         {187562,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                         {187563,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                         {187564,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                         {187565,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}]},
                       1,
                       {queue,[],[],0},
                       {qstate,<0.900.0>,dormant,
                           {1,
                            {<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>,
                             {credit,1,auto},
                             nil,nil}}},
                       168}},
                  {credit_blocked,[]},
                  {{xtype_to_module,direct},rabbit_exchange_type_direct},
                  {{xtype_to_module,topic},rabbit_exchange_type_topic},
                  {{credit_to,<0.552.0>},50},
                  {guid,{{230980502,69406936,3284697005,3000754076},0}},
                  {{credit_from,<0.205.0>},1350}]
    trap_exit: true
    status: running
    heap_size: 1346269
    stack_size: 24
    reductions: 1748356441
  neighbours:

=SUPERVISOR REPORT==== 9-Apr-2014::18:42:05 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    child_terminated
     Reason:     {{badmatch,
                   {error,
                    {misread,
                     [{old_state,
                       {client_msstate,msg_store_persistent,
                        
<<184,251,38,246,192,82,244,149,210,80,115,109,162,244,
                          176,61>>,
                        {dict,1,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],
                           [['...']],
                           [],[],[],[],[],[]}}},
                        {state,233544,"/data/rabbitmq/mnesia/rabbit at pn..."},
                        rabbit_msg_store_ets_index,
                        
"/data/rabbitmq/mnesia/rabbit at pnewellra...",<0.210.0>,
                        237641,229447,241738,245835}},
                      {file_num,681},
                      {offset,16669770},
                      {msg_id,
                      
 <<160,82,77,79,243,175,93,42,1,219,170,168,159,121,222,
                         119>>},
                      {read,
                       {ok,
                        <<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1,
                          219,170,168,159,121,222,119,131,104,6,100,0,13,98,
                          97,115,105,99,95,109,101,115,115,97,103,101,104,4,
                          
100,0,8,114,101,115,111,117,114,99,101,109,0,0,0,2,
                          
100,49,100,0,8,101,120,99,104,97,110,103,101,109,0,
                          0,0,19,66,105,108,108,105,110,103,78,111,116,105,
                          102,105,99,97,116,105,111,110,108,0,0,0,1,109,0,0,
                          
0,10,97,112,112,49,46,100,101,98,117,103,106,104,6,
                          100,0,7,99,111,110,116,101,110,116,97,60,100,0,4,
                          110,111,110,101,109,0,0,0,29,152,0,24,97,112,112,
                          108,105,99,97,116,105,111,110,47,111,99,116,101,
                          116,45,115,116,114,101,97,109,2,0,100,0,25,114,97,
                          98,98,105,116,95,102,114,97,109,105,110,103,95,97,
                          109,113,112,95,48,95,57,95,49,108,0,0,0,1,109,0,0,
                          
40,183,123,34,117,117,105,100,34,58,34,52,48,53,97,
                          98,51,48,97,45,53,57,55,98,45,52,102,97,57,45,57,
                          
101,100,100,45,50,102,100,97,51,56,57,50,57,100,51,
                          50,34,44,34,115,101,113,117,101,110,99,101,34,58,
                          49,56,55,53,54,56,44,34,97,112,112,108,105,99,97,
                          116,105,111,110,34,58,34,97,112,112,48,34,44,34,
                          101,118,101,110,116,34,58,34,69,82,82,79,82,34,44,
                          34,100,97,116,97,34,58,123,34,115,116,114,105,110,
                          103,34,58,34,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                          97,46,46,46>>}},
                      {proc_dict,
                       [{random_seed,{1397,21447,511}},
                        {{"/data/rabbitmq/mnesia/ra...",fhc_file},
                         {file,1,true}},
                        {'$ancestors',
                         [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
                        {process_name,
                         {rabbit_amqqueue_process,
                          {resource,<<"d1">>,queue,<<"Appconnect">>}}},
                        {fhc_age_tree,
                         {2,
                          {{1397,83251,807183},
                           #Ref<0.0.0.1152>,nil,
                          
 {{1397,83251,808277},#Ref<0.0.0.31857>,nil,nil}}}},
                        {{ch,<0.901.0>},
                         {cr,<0.901.0>,#Ref<0.0.0.27418>,
                          {[{187567,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                            {187566,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}],
                           [{187561,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                            {187562,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                            {187563,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
                            {187564,'...'},
                            {'...'}]},
                          1,
                          {queue,[],[],0},
                          {qstate,<0.900.0>,dormant,{1,{'...'}}},
                          168}},
                        {credit_blocked,[]},
                        {{#Ref<0.0.0.1152>,fhc_handle},
                         {handle,
                          {file_descriptor,prim_file,{#Port<0.13248>,15}},
                          0,false,291048,infinity,
                          [[<<192,0,0,0,0,2,220,168>>],
                           [<<128,0,0,0,0,2,220,175>>],
                           [<<192,0,0,0,0,2,220,167>>],
                           [<<128,0,0,0,0,2,220,174>>],
                           [<<192,0,0,0,0,2,220,166>>],
                           [<<128,0,0,0,0,2,220,173>>],
                           [<<192,0,0,0,0,2,220,165>>],
                           ['...'],
                           ['...'],
                           ['...'],
                           ['...'],
                           ['...'],
                           ['...'],
                           ['...'],
                           '...'],
                          true,"/data/rab...",
                          [write,binary,raw,read],
                          [{write_buffer,infinity}],
                          true,true,
                          {1397,83251,807183}}},
                        {{"/data/rabbitmq/mn...",fhc_file},{file,1,false}},
                        {{#Ref<0.0.0.31857>,fhc_handle},
                         {handle,
                          {file_descriptor,prim_file,{#Port<0.18591>,23}},
                          16680435,false,0,1048576,[],false,"/data/r...",
                          [raw,binary,read],
                          [{write_buffer,'...'}],
                          false,true,
                          {1397,83251,808277}}},
                        {{credit_to,<0.552.0>},50},
                        {'$initial_call',{gen,init_it,6}},
                        
{guid,{{230980502,69406936,3284697005,3000754076},0}},
                        {{credit_from,<0.205.0>},1350}]}]}}},
                  [{rabbit_msg_store,read_from_disk,2},
                   {rabbit_msg_store,client_read3,3},
                   {rabbit_msg_store,safe_ets_update_counter,5},
                   {rabbit_variable_queue,with_msg_store_state,3},
                   {rabbit_variable_queue,read_msg,3},
                   {rabbit_variable_queue,fetch,2},
                   {rabbit_mirror_queue_master,fetch,2},
                   {rabbit_amqqueue_process,fetch,2}]}
     Offender:   [{pid,<0.214.0>},
                  {name,rabbit_amqqueue},
                  {mfargs,{rabbit_amqqueue_process,start_link,undefined}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]


=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
  crasher:
    initial call: gen:init_it/7
    pid: <0.205.0>
    registered_name: msg_store_persistent
    exception exit: {{case_clause,
                         {false,
                             {msg_location,
                                
 <<94,44,8,71,59,30,61,222,43,254,204,232,194,
                                   42,158,238>>,
                                 0,681,16584450,10665}}},
                     [{rabbit_msg_store,remove_message,3},
                      {rabbit_msg_store,'-handle_cast/2-fun-0-',3},
                      {lists,foldl,3},
                      {rabbit_msg_store,handle_cast,2},
                      {gen_server2,handle_msg,2},
                      {proc_lib,wake_up,3}]}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_sup,<0.145.0>]
    messages: [{'$gen_cast',{delete_file,1634,16787111}},
                  {'$gen_cast',{delete_file,1654,16787101}},
                  {'$gen_cast',
                   {client_dying,
                    <<184,251,38,246,192,82,244,149,210,80,115,109,162,244,
                      176,61>>}},
                  {'$gen_cast',
                   {client_delete,
                    <<184,251,38,246,192,82,244,149,210,80,115,109,162,244,
                      176,61>>}},
                  {'DOWN',#Ref<0.0.0.1105>,process,<0.214.0>,
                   {{badmatch,
                     {error,
                      {misread,
                       [{old_state,
                         {client_msstate,msg_store_persistent,
                          
<<184,251,38,246,192,82,244,149,210,80,115,109,162,
                            244,176,61>>,
                          {dict,1,'...'},
                          {state,'...'},
                          rabbit_msg_store_ets_index,"...",<0.210.0>,237641,
                          229447,241738,245835}},
                        {file_num,681},
                        {offset,16669770},
                        {msg_id,
                         <<160,82,77,79,243,175,93,42,1,219,170,168,159,121,
                           222,119>>},
                        {read,
                         {ok,
                          <<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1,
                            219,170,168,159,121,222,119,131,104,6,100,0,13,
                            98,97,115,105,99,95,109,101,115,115,97,103,101,
                            
104,4,100,0,8,114,101,115,111,117,114,99,101,109,
                            
0,0,0,2,100,49,100,0,8,101,120,99,104,97,110,103,
                            101,109,0,0,0,19,66,105,108,108,105,110,103,78,
                            111,116,105,102,105,99,97,116,105,111,110,108,0,
                            0,0,1,109,0,0,0,10,97,112,112,49,46,100,101,98,
                            
117,103,106,104,6,100,0,7,99,111,110,116,101,110,
                            116,97,60,100,0,4,110,111,110,101,109,0,0,0,29,
                            152,0,24,97,112,112,108,105,99,97,116,105,111,
                            110,47,111,99,116,101,116,45,115,116,114,101,97,
                            
109,2,0,100,0,25,114,97,98,98,105,116,95,102,114,
                            
97,109,105,110,103,95,97,109,113,112,95,48,95,57,
                            95,49,108,0,0,0,1,109,0,0,40,183,123,34,117,117,
                            105,100,34,58,34,52,48,53,97,98,51,48,97,45,53,
                            
57,55,98,45,52,102,97,57,45,57,101,100,100,45,50,
                            
102,100,97,51,56,57,50,57,100,51,50,34,44,34,115,
                            101,113,117,101,110,99,101,34,58,49,56,55,53,54,
                            56,44,34,97,112,112,108,105,99,97,116,105,111,
                            110,34,58,34,97,112,112,48,34,44,34,101,118,101,
                            110,116,34,58,34,69,82,82,79,82,34,44,34,100,97,
                            116,97,34,58,123,34,115,116,114,105,110,103,34,
                            58,34,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
                            97,97,97,97,97,97,97,46,46,46>>}},
                        {proc_dict,
                         [{'...'},
                          {'...'},
                          {'...'},
                          {'...'},
                          {'...'},
                          {'...'},
                          {'...'},
                          '...']}]}}},
                    [{rabbit_msg_store,read_from_disk,2},
                     {rabbit_msg_store,client_read3,3},
                     {rabbit_msg_store,safe_ets_update_counter,5},
                     {rabbit_variable_queue,with_msg_store_state,3},
                     {rabbit_variable_queue,read_msg,3},
                     {rabbit_variable_queue,fetch,2},
                     {rabbit_mirror_queue_master,fetch,2},
                     {rabbit_amqqueue_process,fetch,2}]}},
                  {'$gen_cast',{delete_file,1653,16787145}},
                  {'$gen_cast',{delete_file,1652,16787106}},
                  {'$gen_cast',{delete_file,1651,16787067}},
                  {'$gen_cast',{delete_file,1650,16787153}},
                  {'$gen_cast',{delete_file,1649,16787131}},
                  {'$gen_cast',{delete_file,1648,16787160}},
                  {'$gen_cast',{delete_file,1647,16787155}},
                  {'$gen_cast',{delete_file,1646,16787099}},
                  {'$gen_cast',{delete_file,1645,16787086}},
                  {'$gen_cast',{delete_file,1665,16787076}},
                  {'$gen_cast',{delete_file,1664,16787088}},
                  {'$gen_cast',{delete_file,1663,16787115}},
                  {'$gen_cast',{delete_file,1662,16787099}},
                  {'$gen_cast',{delete_file,1661,16787074}},
                  {'$gen_cast',{delete_file,1660,16787049}},
                  {'$gen_cast',{delete_file,1659,16787133}},
                  {'$gen_cast',{delete_file,1658,16787128}},
                  {'$gen_cast',{delete_file,1657,16787110}},
                  {'$gen_cast',{delete_file,1656,16787049}},
                  {'$gen_cast',{delete_file,1655,16787126}},
                  {'$gen_cast',{delete_file,1675,16787031}},
                  {'$gen_cast',{delete_file,1674,16787101}},
                  {'$gen_cast',{delete_file,1673,16787040}},
                  {'$gen_cast',{delete_file,1672,16787094}},
                  {'$gen_cast',{delete_file,1671,16787100}},
                  {'$gen_cast',{delete_file,1670,16787017}},
                  {'$gen_cast',{delete_file,1669,16787096}},
                  {'$gen_cast',{delete_file,1668,16787175}},
                  {'$gen_cast',{delete_file,1667,16787143}},
                  {'$gen_cast',{delete_file,1666,16787019}},
                  {'$gen_cast',{delete_file,1686,16787085}},
                  {'$gen_cast',{delete_file,1685,16787079}},
                  {'$gen_cast',{delete_file,1684,16787117}},
                  {'$gen_cast',{delete_file,1683,16787198}},
                  {'$gen_cast',{delete_file,1682,16787109}},
                  {'$gen_cast',{delete_file,1681,16787141}},
                  {'$gen_cast',{delete_file,1680,16787142}},
                  {'$gen_cast',{delete_file,1679,16787100}},
                  {'$gen_cast',{delete_file,1678,16787137}},
                  {'$gen_cast',{delete_file,1677,16787122}},
                  {'$gen_cast',{delete_file,1676,16787092}},
                  {'$gen_cast',{delete_file,1696,16787007}},
                  {'$gen_cast',{delete_file,1695,16787153}},
                  {'$gen_cast',{delete_file,1694,16787025}},
                  {'$gen_cast',{delete_file,1693,16787045}},
                  {'$gen_cast',{delete_file,1692,16787134}},
                  {'$gen_cast',{delete_file,1691,16787109}},
                  {'$gen_cast',{delete_file,1690,16787153}},
                  {'$gen_cast',{delete_file,1689,16787121}},
                  {'$gen_cast',{delete_file,1688,16787109}},
                  {'$gen_cast',{delete_file,1687,16787047}},
                  {'$gen_cast',{delete_file,1706,16787131}},
                  {'$gen_cast',{delete_file,1705,16787087}},
                  {'$gen_cast',{delete_file,1704,16787098}},
                  {'$gen_cast',{delete_file,1703,16787121}},
                  {'$gen_cast',{delete_file,1702,16787086}},
                  {'$gen_cast',{delete_file,1701,16787059}},
                  {'$gen_cast',{delete_file,1700,16787084}},
                  {'$gen_cast',{delete_file,1699,'...'}},
                  {'$gen_cast',{delete_file,'...'}},
                  {'$gen_cast',{'...'}},
                  {'$gen_cast',{'...'}},
                  {'$gen_cast',{'...'}},
                  {'$gen_cast',{'...'}},
                  {'$gen_cast',{'...'}},
                  {'$gen_cast',{'...'}},
                  {'$gen_cast',{'...'}},
                  {'$gen_cast','...'},
                  {'...'},
                  {'...'},
                  {'...'},
                  {'...'},
                  {'...'},
                  {'...'},
                  {'...'},
                  '...']
    links: [<0.146.0>]
    dictionary: [{{credit_to,<0.214.0>},350},{fhc_age_tree,{0,nil}}]
    trap_exit: true
    status: running
    heap_size: 832040
    stack_size: 24
    reductions: 790489983
  neighbours:

=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
     Supervisor: {local,rabbit_sup}
     Context:    child_terminated
     Reason:     {{case_clause,
                      {false,
                          {msg_location,
                              
<<94,44,8,71,59,30,61,222,43,254,204,232,194,42,
                                158,238>>,
                              0,681,16584450,10665}}},
                  [{rabbit_msg_store,remove_message,3},
                   {rabbit_msg_store,'-handle_cast/2-fun-0-',3},
                   {lists,foldl,3},
                   {rabbit_msg_store,handle_cast,2},
                   {gen_server2,handle_msg,2},
                   {proc_lib,wake_up,3}]}
     Offender:   [{pid,<0.205.0>},
                  {name,msg_store_persistent},
                  {mfargs,
                      {rabbit_msg_store,start_link,
                          [msg_store_persistent,
                           "/data/rabbitmq/mnesia/rabbit at pnewellrabbitmqvm",
                           [<<197,113,14,78,169,98,103,74,158,178,5,165,183,
                              174,221,28>>,
                            
<<186,194,196,141,2,108,179,221,99,117,59,65,250,
                              235,145,61>>,
                            
<<184,251,38,246,192,82,244,149,210,80,115,109,162,
                              244,176,61>>,
                            
<<123,55,234,4,244,110,71,11,21,208,154,4,170,7,65,
                              98>>,
                            
<<245,180,254,87,28,109,130,169,124,40,55,231,11,
                              150,52,177>>,
                            <<162,87,146,34,151,178,125,226,157,199,2,24,48,
                              57,175,102>>],
                           {#Fun<rabbit_queue_index.2.111155628>,
                            {start,
                                [{resource,<<"d1-cbs">>,queue,
                                     <<"Appconnect">>},
                                 {resource,<<"/">>,queue,<<"Appconnect">>},
                                 {resource,<<"d1">>,queue,<<"Appconnect">>},
                                
 {resource,<<"d1-cbs">>,queue,<<"ApiUsage">>},
                                 {resource,<<"/">>,queue,<<"ApiUsage">>},
                                 {resource,<<"d1">>,queue,
                                     <<"ApiUsage">>}]}}]}},
                  {restart_type,transient},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
     Supervisor: {local,rabbit_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.205.0>},
                  {name,msg_store_persistent},
                  {mfargs,
                      {rabbit_msg_store,start_link,
                          [msg_store_persistent,
                           "/data/rabbitmq/mnesia/rabbit at pnewellrabbitmqvm",
                           [<<197,113,14,78,169,98,103,74,158,178,5,165,183,
                              174,221,28>>,
                            
<<186,194,196,141,2,108,179,221,99,117,59,65,250,
                              235,145,61>>,
                            
<<184,251,38,246,192,82,244,149,210,80,115,109,162,
                              244,176,61>>,
                            
<<123,55,234,4,244,110,71,11,21,208,154,4,170,7,65,
                              98>>,
                            
<<245,180,254,87,28,109,130,169,124,40,55,231,11,
                              150,52,177>>,
                            <<162,87,146,34,151,178,125,226,157,199,2,24,48,
                              57,175,102>>],
                           {#Fun<rabbit_queue_index.2.111155628>,
                            {start,
                                [{resource,<<"d1-cbs">>,queue,
                                     <<"Appconnect">>},
                                 {resource,<<"/">>,queue,<<"Appconnect">>},
                                 {resource,<<"d1">>,queue,<<"Appconnect">>},
                                
 {resource,<<"d1-cbs">>,queue,<<"ApiUsage">>},
                                 {resource,<<"/">>,queue,<<"ApiUsage">>},
                                 {resource,<<"d1">>,queue,
                                     <<"ApiUsage">>}]}}]}},
                  {restart_type,transient},
                  {shutdown,4294967295},
                  {child_type,worker}]


=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.217.0>
    registered_name: []
    exception exit: {noproc,
                        {gen_server2,call,
                            [msg_store_persistent,
                             {client_terminate,
                                
 <<162,87,146,34,151,178,125,226,157,199,2,24,
                                   48,57,175,102>>},
                             infinity]}}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
    messages: []
    links: [<0.223.0>]
    dictionary: [{random_seed,{1397,21447,29346}},
                  {process_name,
                      {rabbit_amqqueue_process,
                          {resource,<<"d1">>,queue,<<"ApiUsage">>}}},
                  {guid,{{2720735611,2534855728,3608505330,4142421861},0}}]
    trap_exit: true
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 49987
  neighbours:
    neighbour: [{pid,<0.226.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.223.0>,<0.217.0>,rabbit_amqqueue_sup,rabbit_sup,
                       <0.145.0>]},
                  {messages,[]},
                  {links,[<0.223.0>]},
                  {dictionary,
                      [{random_seed,{1397,21447,25352}},
                       {process_name,
                           {gm,{resource,<<"d1">>,queue,<<"ApiUsage">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,233},
                  {stack_size,0},
                  {reductions,244}]
    neighbour: [{pid,<0.223.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.217.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
                  {messages,[]},
                  {links,[<0.217.0>,<0.226.0>]},
                  {dictionary,
                      [{process_name,
                           {rabbit_mirror_queue_coordinator,
                               {resource,<<"d1">>,queue,<<"ApiUsage">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,377},
                  {stack_size,0},
                  {reductions,120}]

=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.216.0>
    registered_name: []
    exception exit: {noproc,
                        {gen_server2,call,
                            [msg_store_persistent,
                             {client_terminate,
                                 <<245,180,254,87,28,109,130,169,124,40,55,
                                   231,11,150,52,177>>},
                             infinity]}}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
    messages: []
    links: [<0.221.0>]
    dictionary: [{random_seed,{1397,21447,28485}},
                  {process_name,
                      {rabbit_amqqueue_process,
                          {resource,<<"/">>,queue,<<"ApiUsage">>}}},
                  {guid,{{2480445846,550087633,2809237486,3458201455},0}}]
    trap_exit: true
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 6301
  neighbours:
    neighbour: [{pid,<0.224.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.221.0>,<0.216.0>,rabbit_amqqueue_sup,rabbit_sup,
                       <0.145.0>]},
                  {messages,[]},
                  {links,[<0.221.0>]},
                  {dictionary,
                      [{random_seed,{1397,21447,25330}},
                       {process_name,
                           {gm,{resource,<<"/">>,queue,<<"ApiUsage">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,233},
                  {stack_size,0},
                  {reductions,244}]
    neighbour: [{pid,<0.221.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.216.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
                  {messages,[]},
                  {links,[<0.216.0>,<0.224.0>]},
                  {dictionary,
                      [{process_name,
                           {rabbit_mirror_queue_coordinator,
                               {resource,<<"/">>,queue,<<"ApiUsage">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,377},
                  {stack_size,0},
                  {reductions,120}]

=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.215.0>
    registered_name: []
    exception exit: {noproc,
                        {gen_server2,call,
                            [msg_store_persistent,
                             {client_terminate,
                                 <<123,55,234,4,244,110,71,11,21,208,154,4,
                                   170,7,65,98>>},
                             infinity]}}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
    messages: []
    links: [<0.222.0>]
    dictionary: [{random_seed,{1397,21447,29365}},
                  {process_name,
                      {rabbit_amqqueue_process,
                          {resource,<<"d1-cbs">>,queue,<<"ApiUsage">>}}},
                  {guid,{{1612108673,2871644604,1399026086,1533985467},0}}]
    trap_exit: true
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 49896
  neighbours:
    neighbour: [{pid,<0.225.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.222.0>,<0.215.0>,rabbit_amqqueue_sup,rabbit_sup,
                       <0.145.0>]},
                  {messages,[]},
                  {links,[<0.222.0>]},
                  {dictionary,
                      [{random_seed,{1397,21447,25344}},
                       {process_name,
                           {gm,{resource,<<"d1-cbs">>,queue,
                                   <<"ApiUsage">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,233},
                  {stack_size,0},
                  {reductions,244}]
    neighbour: [{pid,<0.222.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.215.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
                  {messages,[]},
                  {links,[<0.215.0>,<0.225.0>]},
                  {dictionary,
                      [{process_name,
                           {rabbit_mirror_queue_coordinator,
                               {resource,<<"d1-cbs">>,queue,
                                   <<"ApiUsage">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,377},
                  {stack_size,0},
                  {reductions,120}]

=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.213.0>
    registered_name: []
    exception exit: {noproc,
                        {gen_server2,call,
                            [msg_store_persistent,
                             {client_terminate,
                                
 <<186,194,196,141,2,108,179,221,99,117,59,65,
                                   250,235,145,61>>},
                             infinity]}}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
    messages: []
    links: [<0.227.0>]
    dictionary: [{random_seed,{1397,21447,30135}},
                  {process_name,
                      {rabbit_amqqueue_process,
                          {resource,<<"/">>,queue,<<"Appconnect">>}}},
                  {guid,{{2712885445,3207113857,3507961840,57524982},0}}]
    trap_exit: true
    status: running
    heap_size: 2584
    stack_size: 24
    reductions: 51725
  neighbours:
    neighbour: [{pid,<0.228.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.227.0>,<0.213.0>,rabbit_amqqueue_sup,rabbit_sup,
                       <0.145.0>]},
                  {messages,[]},
                  {links,[<0.227.0>]},
                  {dictionary,
                      [{random_seed,{1397,21447,25502}},
                       {process_name,
                          
 {gm,{resource,<<"/">>,queue,<<"Appconnect">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,233},
                  {stack_size,0},
                  {reductions,245}]
    neighbour: [{pid,<0.227.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.213.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
                  {messages,[]},
                  {links,[<0.213.0>,<0.228.0>]},
                  {dictionary,
                      [{process_name,
                           {rabbit_mirror_queue_coordinator,
                              
 {resource,<<"/">>,queue,<<"Appconnect">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,377},
                  {stack_size,0},
                  {reductions,119}]

=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.212.0>
    registered_name: []
    exception exit: {noproc,
                        {gen_server2,call,
                            [msg_store_persistent,
                             {client_terminate,
                                
 <<197,113,14,78,169,98,103,74,158,178,5,165,
                                   183,174,221,28>>},
                             infinity]}}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
    messages: []
    links: [<0.219.0>]
    dictionary: [{random_seed,{1397,21447,24766}},
                  {process_name,
                      {rabbit_amqqueue_process,
                          {resource,<<"d1-cbs">>,queue,<<"Appconnect">>}}},
                  {guid,{{3043230381,2235775552,2320571086,906396196},0}}]
    trap_exit: true
    status: running
    heap_size: 2584
    stack_size: 24
    reductions: 6321
  neighbours:
    neighbour: [{pid,<0.220.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.219.0>,<0.212.0>,rabbit_amqqueue_sup,rabbit_sup,
                       <0.145.0>]},
                  {messages,[]},
                  {links,[<0.219.0>]},
                  {dictionary,
                      [{random_seed,{1397,21447,23446}},
                       {process_name,
                           {gm,{resource,<<"d1-cbs">>,queue,
                                   <<"Appconnect">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,233},
                  {stack_size,0},
                  {reductions,245}]
    neighbour: [{pid,<0.219.0>},
                  {registered_name,[]},
                  {initial_call,
                      {gen,init_it,
                          ['Argument__1','Argument__2','Argument__3',
                           'Argument__4','Argument__5','Argument__6']}},
                  {current_function,{erlang,hibernate,3}},
                  {ancestors,
                      [<0.212.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
                  {messages,[]},
                  {links,[<0.212.0>,<0.220.0>]},
                  {dictionary,
                      [{process_name,
                           {rabbit_mirror_queue_coordinator,
                               {resource,<<"d1-cbs">>,queue,
                                   <<"Appconnect">>}}}]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,377},
                  {stack_size,0},
                  {reductions,119}]

=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    shutdown_error
     Reason:     {noproc,
                     {gen_server2,call,
                         [msg_store_persistent,
                          {client_terminate,
                              <<186,194,196,141,2,108,179,221,99,117,59,65,
                                250,235,145,61>>},
                          infinity]}}
     Offender:   [{nb_children,1},
                  {name,rabbit_amqqueue},
                  {mfargs,{rabbit_amqqueue_process,start_link,[]}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    shutdown_error
     Reason:     {noproc,
                     {gen_server2,call,
                         [msg_store_persistent,
                          {client_terminate,
                              
<<162,87,146,34,151,178,125,226,157,199,2,24,48,
                                57,175,102>>},
                          infinity]}}
     Offender:   [{nb_children,1},
                  {name,rabbit_amqqueue},
                  {mfargs,{rabbit_amqqueue_process,start_link,[]}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    shutdown_error
     Reason:     {noproc,
                     {gen_server2,call,
                         [msg_store_persistent,
                          {client_terminate,
                              
<<197,113,14,78,169,98,103,74,158,178,5,165,183,
                                174,221,28>>},
                          infinity]}}
     Offender:   [{nb_children,1},
                  {name,rabbit_amqqueue},
                  {mfargs,{rabbit_amqqueue_process,start_link,[]}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    shutdown_error
     Reason:     {noproc,
                     {gen_server2,call,
                         [msg_store_persistent,
                          {client_terminate,
                              <<245,180,254,87,28,109,130,169,124,40,55,231,
                                11,150,52,177>>},
                          infinity]}}
     Offender:   [{nb_children,1},
                  {name,rabbit_amqqueue},
                  {mfargs,{rabbit_amqqueue_process,start_link,[]}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    shutdown_error
     Reason:     {noproc,
                     {gen_server2,call,
                         [msg_store_persistent,
                          {client_terminate,
                              
<<123,55,234,4,244,110,71,11,21,208,154,4,170,7,
                                65,98>>},
                          infinity]}}
     Offender:   [{nb_children,1},
                  {name,rabbit_amqqueue},
                  {mfargs,{rabbit_amqqueue_process,start_link,[]}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]

(This same crash report is repeated several times afterwards)
=CRASH REPORT==== 9-Apr-2014::18:44:08 ===
  crasher:
    initial call: rabbit_mgmt_external_stats:init/1
    pid: <0.248.0>
    registered_name: rabbit_mgmt_external_stats
    exception exit: {noproc,
                        {gen_server,call,
                            [rabbit_node_monitor,partitions,infinity]}}
      in function  gen_server:terminate/6
    ancestors: [rabbit_mgmt_agent_sup,<0.246.0>]
    messages: []
    links: [<0.247.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 375236
  neighbours:


        
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140414/d3aec79c/attachment.html>


More information about the rabbitmq-discuss mailing list