[rabbitmq-discuss] Queue Deleted and Permissions Corrupted While Testing Flow Control
patrickjnewell at gmail.com
patrickjnewell at gmail.com
Mon Apr 14 14:12:32 BST 2014
Hello,
While testing disk-based flow control, the queue my test client was
consuming from suddenly got deleted and the user permissions appeared to
get corrupted. I was running the broker on a CentOS 6.5 VM and had
recently upgraded the broker from 3.2.1 to 3.3.0. The broker had 32 GB of
available disk space with a 48 MB low watermark. The performed the test as
follows:
1) Start a producer that sends 4 million persistent, 10 kb messages to a
durable queue
2) Wait for the producer to block due to disk-based flow control (right
around 3.1 million messages)
3) Start a consumer to drain the queue (prefetch count of 8, acknowledges
one message at a time), allowing the producer to continue to send messages
About 1 minute into step 3, the producer and consumer stopped suddenly and
were disconnected about 1 minute later (the consumer received a Cancel).
The queue disappeared from the management dashboard around the same time.
About 30 seconds later, I was unable to use the dashboard at all -- my
credentials appeared to be invalid and I was seeing 500 errors in the
browser.
The 500 error was as follows:
{error,badarg,
[{gen_event,send,2},
{rabbit_access_control,check_user_login,2},
{rabbit_mgmt_util,is_authorized,6},
{webmachine_resource,resource_call,3},
{webmachine_resource,do,3}
{webmachine_decision_core,resource_call,1},
{webmachine_decision_core,decision,1},
{webmachine_decision_core,handle_request,2}]}
I ran rabbitmqctl and saw that the users and virtual hosts I had created
still existed. However, all exchanges and queues were no longer listed.
Additionally, rabbitmqctl appeared to crash whenever I ran
"list_user_permissions" or "list_permissions":
[root at pnewellrabbitmqvm ~]# rabbitmqctl list_user_permissions admin
Listing permissions for user "admin" ...
Error: {noproc,{gen_server2,call,
[worker_pool,{next_free,<4967.1308.0>},infinity]}}
[root at pnewellrabbitmqvm ~]# rabbitmqctl list_permissions
Listing permissions in vhost "/" ...
Error: {noproc,{gen_server2,call,
[worker_pool,{next_free,<4967.1317.0>},infinity]}}
The sasl.log file listed several crash reports. It looks like the
persistent message store process(?) crashed when it was reading a message
from disk, which caused subsequent crashes. I included the log output
below.
After resetting the broker (rabbitmqctl stop_app, reset, start_app) and
recreating the objects, I ran the same test again and reproduced the same
issue. This time, however, the producer had managed to finish sending its
messages and the consumer had received about 1.3 million messages before
the queue and permissions crashed. I re-ran the same test 3 or 4 more
times, but did not reproduce the issue.
Any idea what might have caused this? My best guess is that something
might have gotten corrupted when I originally performed the upgrade -- I
stopped the service (/sbin/service rabbitmq-server stop), upgraded the
broker and restarted it. The broker was empty with no messages persisted
at the time of the upgrade.
Any help would be much appreciated.
Thanks,
-Pat
Here's the cleaned up sasl-log output (if I took out anything important,
please let me know):
=CRASH REPORT==== 9-Apr-2014::18:42:05 ===
crasher:
initial call: gen:init_it/6
pid: <0.214.0>
registered_name: []
exception exit: {{badmatch,
{error,
{misread,
[{old_state,
{client_msstate,msg_store_persistent,
<<184,251,38,246,192,82,244,149,210,80,115,109,162,
244,176,61>>,
{dict,1,16,16,8,80,48,
{[],[],[],[],'...'},
{{'...'}}},
{state,233544,"/data..."},
rabbit_msg_store_ets_index,"/data/rabbit...",
<0.210.0>,237641,229447,241738,245835}},
{file_num,681},
{offset,16669770},
{msg_id,
<<160,82,77,79,243,175,93,42,1,219,170,168,159,121,
222,119>>},
{read,
{ok,
<<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1,
219,170,168,159,121,222,119,131,104,6,100,0,13,
98,97,115,105,99,95,109,101,115,115,97,103,101,
104,4,100,0,8,114,101,115,111,117,114,99,101,
109,0,0,0,2,100,49,100,0,8,101,120,99,104,97,
110,103,101,109,0,0,0,19,66,105,108,108,105,110,
103,78,111,116,105,102,105,99,97,116,105,111,
110,108,0,0,0,1,109,0,0,0,10,97,112,112,49,46,
100,101,98,117,103,106,104,6,100,0,7,99,111,110,
116,101,110,116,97,60,100,0,4,110,111,110,101,
109,0,0,0,29,152,0,24,97,112,112,108,105,99,97,
116,105,111,110,47,111,99,116,101,116,45,115,
116,114,101,97,109,2,0,100,0,25,114,97,98,98,
105,116,95,102,114,97,109,105,110,103,95,97,109,
113,112,95,48,95,57,95,49,108,0,0,0,1,109,0,0,
40,183,123,34,117,117,105,100,34,58,34,52,48,53,
97,98,51,48,97,45,53,57,55,98,45,52,102,97,57,
45,57,101,100,100,45,50,102,100,97,51,56,57,50,
57,100,51,50,34,44,34,115,101,113,117,101,110,
99,101,34,58,49,56,55,53,54,56,44,34,97,112,112,
108,105,99,97,116,105,111,110,34,58,34,97,112,
112,48,34,44,34,101,118,101,110,116,34,58,34,69,
82,82,79,82,34,44,34,100,97,116,97,34,58,123,34,
115,116,114,105,110,103,34,58,34,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,46,46,46>>}},
{proc_dict,
[{random_seed,{1397,21447,511}},
{{"...",fhc_file},{file,1,true}},
{'$ancestors',
[rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
{process_name,{rabbit_amqqueue_process,{'...'}}},
{fhc_age_tree,{2,{'...'}}},
{{ch,<0.901.0>},
{cr,<0.901.0>,#Ref<0.0.0.27418>,'...'}},
{credit_blocked,[]},
{{#Ref<0.0.0.1152>,fhc_handle},{handle,'...'}},
{{"...",'...'},{'...'}},
{{'...'},{'...'}},
{{'...'},50},
{'$initial_call',{'...'}},
{guid,{'...'}},
{{'...'},1350}]}]}}},
[{rabbit_msg_store,read_from_disk,2},
{rabbit_msg_store,client_read3,3},
{rabbit_msg_store,safe_ets_update_counter,5},
{rabbit_variable_queue,with_msg_store_state,3},
{rabbit_variable_queue,read_msg,3},
{rabbit_variable_queue,fetch,2},
{rabbit_mirror_queue_master,fetch,2},
{rabbit_amqqueue_process,fetch,2}]}
in function gen_server2:terminate/3
ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
messages: [{'$gen_cast',
{deliver,
{delivery,false,false,<0.552.0>,
{basic_message,
{resource,<<"d1">>,exchange,<<"BillingNotification">>},
[<<"app1.debug">>],
{content,60,
{'P_basic',<<"application/octet-stream">>,undefined,
undefined,2,0,undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,undefined},
<<152,0,24,97,112,112,108,105,99,97,116,105,111,110,47,
111,99,116,101,116,45,115,116,114,101,97,109,2,0>>,
rabbit_framing_amqp_0_9_1,
[<<"{\"uuid\":\"dae09ba7-bf47-455c-b512-49d434fb7c3f\",\"sequence\":2363388,\"application\":\"app2\",\"event\":\"DEBUG\",\"data\":{\"string\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...">>]},
<<30,161,133,41,140,102,238,100,34,46,141,56,48,162,217,
14>>,
true},
undefined},
false,flow}},
// Lots of queued up messages, acks, etc...
'...']
links: [<0.211.0>]
dictionary: [{random_seed,{1397,21447,511}},
{process_name,
{rabbit_amqqueue_process,
{resource,<<"d1">>,queue,<<"Appconnect">>}}},
{fhc_age_tree,{0,nil}},
{{ch,<0.901.0>},
{cr,<0.901.0>,#Ref<0.0.0.27418>,
{[{187567,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187566,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}],
[{187561,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187562,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187563,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187564,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187565,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}]},
1,
{queue,[],[],0},
{qstate,<0.900.0>,dormant,
{1,
{<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>,
{credit,1,auto},
nil,nil}}},
168}},
{credit_blocked,[]},
{{xtype_to_module,direct},rabbit_exchange_type_direct},
{{xtype_to_module,topic},rabbit_exchange_type_topic},
{{credit_to,<0.552.0>},50},
{guid,{{230980502,69406936,3284697005,3000754076},0}},
{{credit_from,<0.205.0>},1350}]
trap_exit: true
status: running
heap_size: 1346269
stack_size: 24
reductions: 1748356441
neighbours:
=SUPERVISOR REPORT==== 9-Apr-2014::18:42:05 ===
Supervisor: {local,rabbit_amqqueue_sup}
Context: child_terminated
Reason: {{badmatch,
{error,
{misread,
[{old_state,
{client_msstate,msg_store_persistent,
<<184,251,38,246,192,82,244,149,210,80,115,109,162,244,
176,61>>,
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],
[['...']],
[],[],[],[],[],[]}}},
{state,233544,"/data/rabbitmq/mnesia/rabbit at pn..."},
rabbit_msg_store_ets_index,
"/data/rabbitmq/mnesia/rabbit at pnewellra...",<0.210.0>,
237641,229447,241738,245835}},
{file_num,681},
{offset,16669770},
{msg_id,
<<160,82,77,79,243,175,93,42,1,219,170,168,159,121,222,
119>>},
{read,
{ok,
<<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1,
219,170,168,159,121,222,119,131,104,6,100,0,13,98,
97,115,105,99,95,109,101,115,115,97,103,101,104,4,
100,0,8,114,101,115,111,117,114,99,101,109,0,0,0,2,
100,49,100,0,8,101,120,99,104,97,110,103,101,109,0,
0,0,19,66,105,108,108,105,110,103,78,111,116,105,
102,105,99,97,116,105,111,110,108,0,0,0,1,109,0,0,
0,10,97,112,112,49,46,100,101,98,117,103,106,104,6,
100,0,7,99,111,110,116,101,110,116,97,60,100,0,4,
110,111,110,101,109,0,0,0,29,152,0,24,97,112,112,
108,105,99,97,116,105,111,110,47,111,99,116,101,
116,45,115,116,114,101,97,109,2,0,100,0,25,114,97,
98,98,105,116,95,102,114,97,109,105,110,103,95,97,
109,113,112,95,48,95,57,95,49,108,0,0,0,1,109,0,0,
40,183,123,34,117,117,105,100,34,58,34,52,48,53,97,
98,51,48,97,45,53,57,55,98,45,52,102,97,57,45,57,
101,100,100,45,50,102,100,97,51,56,57,50,57,100,51,
50,34,44,34,115,101,113,117,101,110,99,101,34,58,
49,56,55,53,54,56,44,34,97,112,112,108,105,99,97,
116,105,111,110,34,58,34,97,112,112,48,34,44,34,
101,118,101,110,116,34,58,34,69,82,82,79,82,34,44,
34,100,97,116,97,34,58,123,34,115,116,114,105,110,
103,34,58,34,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,46,46,46>>}},
{proc_dict,
[{random_seed,{1397,21447,511}},
{{"/data/rabbitmq/mnesia/ra...",fhc_file},
{file,1,true}},
{'$ancestors',
[rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
{process_name,
{rabbit_amqqueue_process,
{resource,<<"d1">>,queue,<<"Appconnect">>}}},
{fhc_age_tree,
{2,
{{1397,83251,807183},
#Ref<0.0.0.1152>,nil,
{{1397,83251,808277},#Ref<0.0.0.31857>,nil,nil}}}},
{{ch,<0.901.0>},
{cr,<0.901.0>,#Ref<0.0.0.27418>,
{[{187567,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187566,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}],
[{187561,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187562,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187563,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>},
{187564,'...'},
{'...'}]},
1,
{queue,[],[],0},
{qstate,<0.900.0>,dormant,{1,{'...'}}},
168}},
{credit_blocked,[]},
{{#Ref<0.0.0.1152>,fhc_handle},
{handle,
{file_descriptor,prim_file,{#Port<0.13248>,15}},
0,false,291048,infinity,
[[<<192,0,0,0,0,2,220,168>>],
[<<128,0,0,0,0,2,220,175>>],
[<<192,0,0,0,0,2,220,167>>],
[<<128,0,0,0,0,2,220,174>>],
[<<192,0,0,0,0,2,220,166>>],
[<<128,0,0,0,0,2,220,173>>],
[<<192,0,0,0,0,2,220,165>>],
['...'],
['...'],
['...'],
['...'],
['...'],
['...'],
['...'],
'...'],
true,"/data/rab...",
[write,binary,raw,read],
[{write_buffer,infinity}],
true,true,
{1397,83251,807183}}},
{{"/data/rabbitmq/mn...",fhc_file},{file,1,false}},
{{#Ref<0.0.0.31857>,fhc_handle},
{handle,
{file_descriptor,prim_file,{#Port<0.18591>,23}},
16680435,false,0,1048576,[],false,"/data/r...",
[raw,binary,read],
[{write_buffer,'...'}],
false,true,
{1397,83251,808277}}},
{{credit_to,<0.552.0>},50},
{'$initial_call',{gen,init_it,6}},
{guid,{{230980502,69406936,3284697005,3000754076},0}},
{{credit_from,<0.205.0>},1350}]}]}}},
[{rabbit_msg_store,read_from_disk,2},
{rabbit_msg_store,client_read3,3},
{rabbit_msg_store,safe_ets_update_counter,5},
{rabbit_variable_queue,with_msg_store_state,3},
{rabbit_variable_queue,read_msg,3},
{rabbit_variable_queue,fetch,2},
{rabbit_mirror_queue_master,fetch,2},
{rabbit_amqqueue_process,fetch,2}]}
Offender: [{pid,<0.214.0>},
{name,rabbit_amqqueue},
{mfargs,{rabbit_amqqueue_process,start_link,undefined}},
{restart_type,temporary},
{shutdown,4294967295},
{child_type,worker}]
=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
crasher:
initial call: gen:init_it/7
pid: <0.205.0>
registered_name: msg_store_persistent
exception exit: {{case_clause,
{false,
{msg_location,
<<94,44,8,71,59,30,61,222,43,254,204,232,194,
42,158,238>>,
0,681,16584450,10665}}},
[{rabbit_msg_store,remove_message,3},
{rabbit_msg_store,'-handle_cast/2-fun-0-',3},
{lists,foldl,3},
{rabbit_msg_store,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}
in function gen_server2:terminate/3
ancestors: [rabbit_sup,<0.145.0>]
messages: [{'$gen_cast',{delete_file,1634,16787111}},
{'$gen_cast',{delete_file,1654,16787101}},
{'$gen_cast',
{client_dying,
<<184,251,38,246,192,82,244,149,210,80,115,109,162,244,
176,61>>}},
{'$gen_cast',
{client_delete,
<<184,251,38,246,192,82,244,149,210,80,115,109,162,244,
176,61>>}},
{'DOWN',#Ref<0.0.0.1105>,process,<0.214.0>,
{{badmatch,
{error,
{misread,
[{old_state,
{client_msstate,msg_store_persistent,
<<184,251,38,246,192,82,244,149,210,80,115,109,162,
244,176,61>>,
{dict,1,'...'},
{state,'...'},
rabbit_msg_store_ets_index,"...",<0.210.0>,237641,
229447,241738,245835}},
{file_num,681},
{offset,16669770},
{msg_id,
<<160,82,77,79,243,175,93,42,1,219,170,168,159,121,
222,119>>},
{read,
{ok,
<<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1,
219,170,168,159,121,222,119,131,104,6,100,0,13,
98,97,115,105,99,95,109,101,115,115,97,103,101,
104,4,100,0,8,114,101,115,111,117,114,99,101,109,
0,0,0,2,100,49,100,0,8,101,120,99,104,97,110,103,
101,109,0,0,0,19,66,105,108,108,105,110,103,78,
111,116,105,102,105,99,97,116,105,111,110,108,0,
0,0,1,109,0,0,0,10,97,112,112,49,46,100,101,98,
117,103,106,104,6,100,0,7,99,111,110,116,101,110,
116,97,60,100,0,4,110,111,110,101,109,0,0,0,29,
152,0,24,97,112,112,108,105,99,97,116,105,111,
110,47,111,99,116,101,116,45,115,116,114,101,97,
109,2,0,100,0,25,114,97,98,98,105,116,95,102,114,
97,109,105,110,103,95,97,109,113,112,95,48,95,57,
95,49,108,0,0,0,1,109,0,0,40,183,123,34,117,117,
105,100,34,58,34,52,48,53,97,98,51,48,97,45,53,
57,55,98,45,52,102,97,57,45,57,101,100,100,45,50,
102,100,97,51,56,57,50,57,100,51,50,34,44,34,115,
101,113,117,101,110,99,101,34,58,49,56,55,53,54,
56,44,34,97,112,112,108,105,99,97,116,105,111,
110,34,58,34,97,112,112,48,34,44,34,101,118,101,
110,116,34,58,34,69,82,82,79,82,34,44,34,100,97,
116,97,34,58,123,34,115,116,114,105,110,103,34,
58,34,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,
97,97,97,97,97,97,97,46,46,46>>}},
{proc_dict,
[{'...'},
{'...'},
{'...'},
{'...'},
{'...'},
{'...'},
{'...'},
'...']}]}}},
[{rabbit_msg_store,read_from_disk,2},
{rabbit_msg_store,client_read3,3},
{rabbit_msg_store,safe_ets_update_counter,5},
{rabbit_variable_queue,with_msg_store_state,3},
{rabbit_variable_queue,read_msg,3},
{rabbit_variable_queue,fetch,2},
{rabbit_mirror_queue_master,fetch,2},
{rabbit_amqqueue_process,fetch,2}]}},
{'$gen_cast',{delete_file,1653,16787145}},
{'$gen_cast',{delete_file,1652,16787106}},
{'$gen_cast',{delete_file,1651,16787067}},
{'$gen_cast',{delete_file,1650,16787153}},
{'$gen_cast',{delete_file,1649,16787131}},
{'$gen_cast',{delete_file,1648,16787160}},
{'$gen_cast',{delete_file,1647,16787155}},
{'$gen_cast',{delete_file,1646,16787099}},
{'$gen_cast',{delete_file,1645,16787086}},
{'$gen_cast',{delete_file,1665,16787076}},
{'$gen_cast',{delete_file,1664,16787088}},
{'$gen_cast',{delete_file,1663,16787115}},
{'$gen_cast',{delete_file,1662,16787099}},
{'$gen_cast',{delete_file,1661,16787074}},
{'$gen_cast',{delete_file,1660,16787049}},
{'$gen_cast',{delete_file,1659,16787133}},
{'$gen_cast',{delete_file,1658,16787128}},
{'$gen_cast',{delete_file,1657,16787110}},
{'$gen_cast',{delete_file,1656,16787049}},
{'$gen_cast',{delete_file,1655,16787126}},
{'$gen_cast',{delete_file,1675,16787031}},
{'$gen_cast',{delete_file,1674,16787101}},
{'$gen_cast',{delete_file,1673,16787040}},
{'$gen_cast',{delete_file,1672,16787094}},
{'$gen_cast',{delete_file,1671,16787100}},
{'$gen_cast',{delete_file,1670,16787017}},
{'$gen_cast',{delete_file,1669,16787096}},
{'$gen_cast',{delete_file,1668,16787175}},
{'$gen_cast',{delete_file,1667,16787143}},
{'$gen_cast',{delete_file,1666,16787019}},
{'$gen_cast',{delete_file,1686,16787085}},
{'$gen_cast',{delete_file,1685,16787079}},
{'$gen_cast',{delete_file,1684,16787117}},
{'$gen_cast',{delete_file,1683,16787198}},
{'$gen_cast',{delete_file,1682,16787109}},
{'$gen_cast',{delete_file,1681,16787141}},
{'$gen_cast',{delete_file,1680,16787142}},
{'$gen_cast',{delete_file,1679,16787100}},
{'$gen_cast',{delete_file,1678,16787137}},
{'$gen_cast',{delete_file,1677,16787122}},
{'$gen_cast',{delete_file,1676,16787092}},
{'$gen_cast',{delete_file,1696,16787007}},
{'$gen_cast',{delete_file,1695,16787153}},
{'$gen_cast',{delete_file,1694,16787025}},
{'$gen_cast',{delete_file,1693,16787045}},
{'$gen_cast',{delete_file,1692,16787134}},
{'$gen_cast',{delete_file,1691,16787109}},
{'$gen_cast',{delete_file,1690,16787153}},
{'$gen_cast',{delete_file,1689,16787121}},
{'$gen_cast',{delete_file,1688,16787109}},
{'$gen_cast',{delete_file,1687,16787047}},
{'$gen_cast',{delete_file,1706,16787131}},
{'$gen_cast',{delete_file,1705,16787087}},
{'$gen_cast',{delete_file,1704,16787098}},
{'$gen_cast',{delete_file,1703,16787121}},
{'$gen_cast',{delete_file,1702,16787086}},
{'$gen_cast',{delete_file,1701,16787059}},
{'$gen_cast',{delete_file,1700,16787084}},
{'$gen_cast',{delete_file,1699,'...'}},
{'$gen_cast',{delete_file,'...'}},
{'$gen_cast',{'...'}},
{'$gen_cast',{'...'}},
{'$gen_cast',{'...'}},
{'$gen_cast',{'...'}},
{'$gen_cast',{'...'}},
{'$gen_cast',{'...'}},
{'$gen_cast',{'...'}},
{'$gen_cast','...'},
{'...'},
{'...'},
{'...'},
{'...'},
{'...'},
{'...'},
{'...'},
'...']
links: [<0.146.0>]
dictionary: [{{credit_to,<0.214.0>},350},{fhc_age_tree,{0,nil}}]
trap_exit: true
status: running
heap_size: 832040
stack_size: 24
reductions: 790489983
neighbours:
=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
Supervisor: {local,rabbit_sup}
Context: child_terminated
Reason: {{case_clause,
{false,
{msg_location,
<<94,44,8,71,59,30,61,222,43,254,204,232,194,42,
158,238>>,
0,681,16584450,10665}}},
[{rabbit_msg_store,remove_message,3},
{rabbit_msg_store,'-handle_cast/2-fun-0-',3},
{lists,foldl,3},
{rabbit_msg_store,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}
Offender: [{pid,<0.205.0>},
{name,msg_store_persistent},
{mfargs,
{rabbit_msg_store,start_link,
[msg_store_persistent,
"/data/rabbitmq/mnesia/rabbit at pnewellrabbitmqvm",
[<<197,113,14,78,169,98,103,74,158,178,5,165,183,
174,221,28>>,
<<186,194,196,141,2,108,179,221,99,117,59,65,250,
235,145,61>>,
<<184,251,38,246,192,82,244,149,210,80,115,109,162,
244,176,61>>,
<<123,55,234,4,244,110,71,11,21,208,154,4,170,7,65,
98>>,
<<245,180,254,87,28,109,130,169,124,40,55,231,11,
150,52,177>>,
<<162,87,146,34,151,178,125,226,157,199,2,24,48,
57,175,102>>],
{#Fun<rabbit_queue_index.2.111155628>,
{start,
[{resource,<<"d1-cbs">>,queue,
<<"Appconnect">>},
{resource,<<"/">>,queue,<<"Appconnect">>},
{resource,<<"d1">>,queue,<<"Appconnect">>},
{resource,<<"d1-cbs">>,queue,<<"ApiUsage">>},
{resource,<<"/">>,queue,<<"ApiUsage">>},
{resource,<<"d1">>,queue,
<<"ApiUsage">>}]}}]}},
{restart_type,transient},
{shutdown,4294967295},
{child_type,worker}]
=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
Supervisor: {local,rabbit_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [{pid,<0.205.0>},
{name,msg_store_persistent},
{mfargs,
{rabbit_msg_store,start_link,
[msg_store_persistent,
"/data/rabbitmq/mnesia/rabbit at pnewellrabbitmqvm",
[<<197,113,14,78,169,98,103,74,158,178,5,165,183,
174,221,28>>,
<<186,194,196,141,2,108,179,221,99,117,59,65,250,
235,145,61>>,
<<184,251,38,246,192,82,244,149,210,80,115,109,162,
244,176,61>>,
<<123,55,234,4,244,110,71,11,21,208,154,4,170,7,65,
98>>,
<<245,180,254,87,28,109,130,169,124,40,55,231,11,
150,52,177>>,
<<162,87,146,34,151,178,125,226,157,199,2,24,48,
57,175,102>>],
{#Fun<rabbit_queue_index.2.111155628>,
{start,
[{resource,<<"d1-cbs">>,queue,
<<"Appconnect">>},
{resource,<<"/">>,queue,<<"Appconnect">>},
{resource,<<"d1">>,queue,<<"Appconnect">>},
{resource,<<"d1-cbs">>,queue,<<"ApiUsage">>},
{resource,<<"/">>,queue,<<"ApiUsage">>},
{resource,<<"d1">>,queue,
<<"ApiUsage">>}]}}]}},
{restart_type,transient},
{shutdown,4294967295},
{child_type,worker}]
=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
crasher:
initial call: gen:init_it/6
pid: <0.217.0>
registered_name: []
exception exit: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<162,87,146,34,151,178,125,226,157,199,2,24,
48,57,175,102>>},
infinity]}}
in function gen_server2:terminate/3
ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
messages: []
links: [<0.223.0>]
dictionary: [{random_seed,{1397,21447,29346}},
{process_name,
{rabbit_amqqueue_process,
{resource,<<"d1">>,queue,<<"ApiUsage">>}}},
{guid,{{2720735611,2534855728,3608505330,4142421861},0}}]
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 49987
neighbours:
neighbour: [{pid,<0.226.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.223.0>,<0.217.0>,rabbit_amqqueue_sup,rabbit_sup,
<0.145.0>]},
{messages,[]},
{links,[<0.223.0>]},
{dictionary,
[{random_seed,{1397,21447,25352}},
{process_name,
{gm,{resource,<<"d1">>,queue,<<"ApiUsage">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,233},
{stack_size,0},
{reductions,244}]
neighbour: [{pid,<0.223.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.217.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
{messages,[]},
{links,[<0.217.0>,<0.226.0>]},
{dictionary,
[{process_name,
{rabbit_mirror_queue_coordinator,
{resource,<<"d1">>,queue,<<"ApiUsage">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,377},
{stack_size,0},
{reductions,120}]
=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
crasher:
initial call: gen:init_it/6
pid: <0.216.0>
registered_name: []
exception exit: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<245,180,254,87,28,109,130,169,124,40,55,
231,11,150,52,177>>},
infinity]}}
in function gen_server2:terminate/3
ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
messages: []
links: [<0.221.0>]
dictionary: [{random_seed,{1397,21447,28485}},
{process_name,
{rabbit_amqqueue_process,
{resource,<<"/">>,queue,<<"ApiUsage">>}}},
{guid,{{2480445846,550087633,2809237486,3458201455},0}}]
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 6301
neighbours:
neighbour: [{pid,<0.224.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.221.0>,<0.216.0>,rabbit_amqqueue_sup,rabbit_sup,
<0.145.0>]},
{messages,[]},
{links,[<0.221.0>]},
{dictionary,
[{random_seed,{1397,21447,25330}},
{process_name,
{gm,{resource,<<"/">>,queue,<<"ApiUsage">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,233},
{stack_size,0},
{reductions,244}]
neighbour: [{pid,<0.221.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.216.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
{messages,[]},
{links,[<0.216.0>,<0.224.0>]},
{dictionary,
[{process_name,
{rabbit_mirror_queue_coordinator,
{resource,<<"/">>,queue,<<"ApiUsage">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,377},
{stack_size,0},
{reductions,120}]
=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
crasher:
initial call: gen:init_it/6
pid: <0.215.0>
registered_name: []
exception exit: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<123,55,234,4,244,110,71,11,21,208,154,4,
170,7,65,98>>},
infinity]}}
in function gen_server2:terminate/3
ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
messages: []
links: [<0.222.0>]
dictionary: [{random_seed,{1397,21447,29365}},
{process_name,
{rabbit_amqqueue_process,
{resource,<<"d1-cbs">>,queue,<<"ApiUsage">>}}},
{guid,{{1612108673,2871644604,1399026086,1533985467},0}}]
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 49896
neighbours:
neighbour: [{pid,<0.225.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.222.0>,<0.215.0>,rabbit_amqqueue_sup,rabbit_sup,
<0.145.0>]},
{messages,[]},
{links,[<0.222.0>]},
{dictionary,
[{random_seed,{1397,21447,25344}},
{process_name,
{gm,{resource,<<"d1-cbs">>,queue,
<<"ApiUsage">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,233},
{stack_size,0},
{reductions,244}]
neighbour: [{pid,<0.222.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.215.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
{messages,[]},
{links,[<0.215.0>,<0.225.0>]},
{dictionary,
[{process_name,
{rabbit_mirror_queue_coordinator,
{resource,<<"d1-cbs">>,queue,
<<"ApiUsage">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,377},
{stack_size,0},
{reductions,120}]
=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
crasher:
initial call: gen:init_it/6
pid: <0.213.0>
registered_name: []
exception exit: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<186,194,196,141,2,108,179,221,99,117,59,65,
250,235,145,61>>},
infinity]}}
in function gen_server2:terminate/3
ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
messages: []
links: [<0.227.0>]
dictionary: [{random_seed,{1397,21447,30135}},
{process_name,
{rabbit_amqqueue_process,
{resource,<<"/">>,queue,<<"Appconnect">>}}},
{guid,{{2712885445,3207113857,3507961840,57524982},0}}]
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 51725
neighbours:
neighbour: [{pid,<0.228.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.227.0>,<0.213.0>,rabbit_amqqueue_sup,rabbit_sup,
<0.145.0>]},
{messages,[]},
{links,[<0.227.0>]},
{dictionary,
[{random_seed,{1397,21447,25502}},
{process_name,
{gm,{resource,<<"/">>,queue,<<"Appconnect">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,233},
{stack_size,0},
{reductions,245}]
neighbour: [{pid,<0.227.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.213.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
{messages,[]},
{links,[<0.213.0>,<0.228.0>]},
{dictionary,
[{process_name,
{rabbit_mirror_queue_coordinator,
{resource,<<"/">>,queue,<<"Appconnect">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,377},
{stack_size,0},
{reductions,119}]
=CRASH REPORT==== 9-Apr-2014::18:44:04 ===
crasher:
initial call: gen:init_it/6
pid: <0.212.0>
registered_name: []
exception exit: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<197,113,14,78,169,98,103,74,158,178,5,165,
183,174,221,28>>},
infinity]}}
in function gen_server2:terminate/3
ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]
messages: []
links: [<0.219.0>]
dictionary: [{random_seed,{1397,21447,24766}},
{process_name,
{rabbit_amqqueue_process,
{resource,<<"d1-cbs">>,queue,<<"Appconnect">>}}},
{guid,{{3043230381,2235775552,2320571086,906396196},0}}]
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 6321
neighbours:
neighbour: [{pid,<0.220.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.219.0>,<0.212.0>,rabbit_amqqueue_sup,rabbit_sup,
<0.145.0>]},
{messages,[]},
{links,[<0.219.0>]},
{dictionary,
[{random_seed,{1397,21447,23446}},
{process_name,
{gm,{resource,<<"d1-cbs">>,queue,
<<"Appconnect">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,233},
{stack_size,0},
{reductions,245}]
neighbour: [{pid,<0.219.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
['Argument__1','Argument__2','Argument__3',
'Argument__4','Argument__5','Argument__6']}},
{current_function,{erlang,hibernate,3}},
{ancestors,
[<0.212.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]},
{messages,[]},
{links,[<0.212.0>,<0.220.0>]},
{dictionary,
[{process_name,
{rabbit_mirror_queue_coordinator,
{resource,<<"d1-cbs">>,queue,
<<"Appconnect">>}}}]},
{trap_exit,false},
{status,waiting},
{heap_size,377},
{stack_size,0},
{reductions,119}]
=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
Supervisor: {local,rabbit_amqqueue_sup}
Context: shutdown_error
Reason: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<186,194,196,141,2,108,179,221,99,117,59,65,
250,235,145,61>>},
infinity]}}
Offender: [{nb_children,1},
{name,rabbit_amqqueue},
{mfargs,{rabbit_amqqueue_process,start_link,[]}},
{restart_type,temporary},
{shutdown,4294967295},
{child_type,worker}]
=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
Supervisor: {local,rabbit_amqqueue_sup}
Context: shutdown_error
Reason: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<162,87,146,34,151,178,125,226,157,199,2,24,48,
57,175,102>>},
infinity]}}
Offender: [{nb_children,1},
{name,rabbit_amqqueue},
{mfargs,{rabbit_amqqueue_process,start_link,[]}},
{restart_type,temporary},
{shutdown,4294967295},
{child_type,worker}]
=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
Supervisor: {local,rabbit_amqqueue_sup}
Context: shutdown_error
Reason: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<197,113,14,78,169,98,103,74,158,178,5,165,183,
174,221,28>>},
infinity]}}
Offender: [{nb_children,1},
{name,rabbit_amqqueue},
{mfargs,{rabbit_amqqueue_process,start_link,[]}},
{restart_type,temporary},
{shutdown,4294967295},
{child_type,worker}]
=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
Supervisor: {local,rabbit_amqqueue_sup}
Context: shutdown_error
Reason: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<245,180,254,87,28,109,130,169,124,40,55,231,
11,150,52,177>>},
infinity]}}
Offender: [{nb_children,1},
{name,rabbit_amqqueue},
{mfargs,{rabbit_amqqueue_process,start_link,[]}},
{restart_type,temporary},
{shutdown,4294967295},
{child_type,worker}]
=SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 ===
Supervisor: {local,rabbit_amqqueue_sup}
Context: shutdown_error
Reason: {noproc,
{gen_server2,call,
[msg_store_persistent,
{client_terminate,
<<123,55,234,4,244,110,71,11,21,208,154,4,170,7,
65,98>>},
infinity]}}
Offender: [{nb_children,1},
{name,rabbit_amqqueue},
{mfargs,{rabbit_amqqueue_process,start_link,[]}},
{restart_type,temporary},
{shutdown,4294967295},
{child_type,worker}]
(This same crash report is repeated several times afterwards)
=CRASH REPORT==== 9-Apr-2014::18:44:08 ===
crasher:
initial call: rabbit_mgmt_external_stats:init/1
pid: <0.248.0>
registered_name: rabbit_mgmt_external_stats
exception exit: {noproc,
{gen_server,call,
[rabbit_node_monitor,partitions,infinity]}}
in function gen_server:terminate/6
ancestors: [rabbit_mgmt_agent_sup,<0.246.0>]
messages: []
links: [<0.247.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 1597
stack_size: 24
reductions: 375236
neighbours:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140414/d3aec79c/attachment.html>
More information about the rabbitmq-discuss
mailing list