[rabbitmq-discuss] Crash - Total persistent queue content loss - 2.5.1
Joseph Marlin
joseph.a.marlin at gmail.com
Mon Apr 2 21:15:22 BST 2012
I know I'm working on a version that's pretty old by now. I looked at the
log contents and was completely mystified. I'd like to understand what went
wrong here.
The Story: RabbitMQ crashes and when I bring it back up, all queue contents
from all my persistent queues are no longer there.
-If it was my fault, what can I do to fix it?
-If it is Rabbit's fault, has the issue been fixed?
Attached are the 'rabbitmqctl report' and the appropriate log from
/var/log/rabbitmq/
Thanks so much!
Joseph Marlin
--------------------------------
Highlights (in case it is easy enough to diagnose without even needing to
download anything:
--------------------------------
****Report****:
Status of node 'rabbit at tile-render' ...
[{pid,29076},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","2.5.1"},
{webmachine,"webmachine","1.7.0-rmq2.5.1-hg0c4b60a"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","2.5.1"},
{amqp_client,"RabbitMQ AMQP Client","2.5.1"},
{rabbit,"RabbitMQ","2.5.1"},
{os_mon,"CPO CXC 138 46","2.2.4"},
{sasl,"SASL CXC 138 11","2.1.8"},
{rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.5.1"},
{mochiweb,"MochiMedia Web Server","1.3-rmq2.5.1-git9a53dbd"},
{inets,"INETS CXC 138 49","5.2"},
{mnesia,"MNESIA CXC 138 12","4.4.12"},
{stdlib,"ERTS CXC 138 10","1.16.4"},
{kernel,"ERTS CXC 138 10","2.13.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8]
[async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,31325888},
{processes,11455384},
{processes_used,11434488},
{system,19870504},
{atom,1269865},
{atom_used,1263088},
{binary,76048},
{code,14856883},
{ets,1091928}]}]
****Log****:
** Reason for termination ==
** {{case_clause,undefined},
[{file_handle_cache,'-partition_handles/1-fun-0-',2},
{file_handle_cache,get_or_reopen,1},
{file_handle_cache,with_handles,2},
{rabbit_msg_store,internal_sync,1},
{rabbit_msg_store,terminate,2},
{gen_server2,terminate,3},
{proc_lib,wake_up,3}]}
** In 'terminate' callback with reason ==
** {{badmatch,{error,enospc}},
[{rabbit_msg_store,maybe_roll_to_new_file,2},
{file_handle_cache,open,3},
{rabbit_msg_store,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,wake_up,3}]}
--truncated--
** Reason for termination ==
** {badarg,[{ets,delete,
[245834,
{<<52,76,19,5,248,12,21,12,214,129,172,73,116,247,183,106>>,
14797}]},
{rabbit_msg_store,'-close_all_handles/1-fun-0-',5},
{dict,fold_bucket,3},
{dict,fold_seg,4},
{dict,fold_segs,4},
{rabbit_msg_store,close_all_handles,1},
{rabbit_msg_store,client_delete_and_terminate,1},
{rabbit_variable_queue,delete_and_terminate,2}]}
** In 'terminate' callback with reason ==
** {badarg,[{ets,lookup,
[249931,
<<167,42,100,148,97,183,72,206,176,227,110,240,207,15,2,93>>]},
{rabbit_msg_store,read,2},
{rabbit_variable_queue,with_msg_store_state,3},
{rabbit_variable_queue,read_msg,2},
{rabbit_variable_queue,'-fetch/2-fun-0-',3},
{rabbit_amqqueue_process,fetch,2},
{rabbit_amqqueue_process,deliver_from_queue_deliver,3},
{rabbit_amqqueue_process,deliver_msgs_to_consumers,3}]}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120402/0cac3daf/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit at tile-render.log.2.gz
Type: application/octet-stream
Size: 1002643 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120402/0cac3daf/attachment-0001.obj>
-------------- next part --------------
Status of node 'rabbit at tile-render' ...
[{pid,29076},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","2.5.1"},
{webmachine,"webmachine","1.7.0-rmq2.5.1-hg0c4b60a"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","2.5.1"},
{amqp_client,"RabbitMQ AMQP Client","2.5.1"},
{rabbit,"RabbitMQ","2.5.1"},
{os_mon,"CPO CXC 138 46","2.2.4"},
{sasl,"SASL CXC 138 11","2.1.8"},
{rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.5.1"},
{mochiweb,"MochiMedia Web Server","1.3-rmq2.5.1-git9a53dbd"},
{inets,"INETS CXC 138 49","5.2"},
{mnesia,"MNESIA CXC 138 12","4.4.12"},
{stdlib,"ERTS CXC 138 10","1.16.4"},
{kernel,"ERTS CXC 138 10","2.13.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,31325888},
{processes,11455384},
{processes_used,11434488},
{system,19870504},
{atom,1269865},
{atom_used,1263088},
{binary,76048},
{code,14856883},
{ets,1091928}]}]
Cluster status of node 'rabbit at tile-render' ...
[{nodes,[{disc,['rabbit at tile-render']}]},
{running_nodes,['rabbit at tile-render']}]
Application environment of node 'rabbit at tile-render' ...
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue},
{cluster_nodes,[]},
{collect_statistics,fine},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_is_admin,true},
{default_vhost,<<"/">>},
{delegate_count,16},
{frame_max,131072},
{included_applications,[]},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{persister_hibernate_after,10000},
{persister_max_wrap_entries,500},
{queue_index_max_journal_entries,262144},
{server_properties,[]},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{exit_on_close,false}]},
{tcp_listeners,[5672]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.4}]
Connections:
Channels:
Queues on /:
pid name durable auto_delete arguments owner_pid exclusive_consumer_pid exclusive_consumer_tag messages_ready messages_unacknowledged messages consumers memory backing_queue_status
<'rabbit at tile-render'.1.236.0> TDS_FROM_DEVICE_QUEUE true false [] 0 0 0 0 34624[{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
<'rabbit at tile-render'.1.237.0> TILES_TO_SAVE true false [] 0 0 0 0 34624 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
<'rabbit at tile-render'.1.238.0> TILES_TO_CREATE true false [] 0 0 0 0 34624 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
Exchanges on /:
name type durable auto_delete internal arguments
amq.direct direct true false false []
amq.topic topic true false false []
amq.rabbitmq.trace topic true false false []
amq.rabbitmq.log topic true false false []
amq.fanout fanout true false false []
submgr.fromDevice direct true false false []
amq.headers headers true false false []
direct true false false []
amq.match headers true false false []
Bindings on /:
source_name source_kind destination_name destination_kind routing_key arguments
exchange TDS_FROM_DEVICE_QUEUE queue TDS_FROM_DEVICE_QUEUE []
exchange TILES_TO_CREATE queue TILES_TO_CREATE []
exchange TILES_TO_SAVE queue TILES_TO_SAVE []
Consumers on /:
Permissions on /:
user configure write read
guest .* .* .*
End of server status report
...done.
More information about the rabbitmq-discuss
mailing list