<div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial">Hi,<div><br></div><div>    I have a problem with rabbitmq recently . </div><div>    The message in queue  accumulated and the queue become more and more bigger(about hundred thousands of messages in queue). Then I found rabbitmq did not push message to consumers and all consumers were blocked on tcp receive function. </div><div>    While the publishers were continue publishing message at that time. (When I found the channel was blocked, I would try to re-create a connection to publish message. Am I wrong  to publish message in such a way?)</div><div>    Is stopping pushing message to consumers the reason of message accumulated or the result of message accumulated? Another thing, the message is durable, and disk io at that time was almost full. May this be the reason of message accumulated? Then, still, why consumers could not receive message from rabbitmq?</div><div>    Can anyone help? Thanks very much in advanced!</div><div>    Here is the status of rabbitmq I dump when problem occur:</div><div><div>Reporting server status on {{2014,4,23},{14,50,1}}</div><div><br></div><div> ...</div><div>Status of node rabbit4@rabbitmq153 ...</div><div>[{pid,8934},</div><div> {running_applications,</div><div>     [{rabbitmq_federation_management,"RabbitMQ Federation Management",</div><div>          "3.1.5"},</div><div>      {rabbitmq_management,"RabbitMQ Management Console","3.1.5"},</div><div>      {rabbitmq_federation,"RabbitMQ Federation","3.1.5"},</div><div>      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.5"},</div><div>      {rabbit,"RabbitMQ","3.1.5"},</div><div>      {os_mon,"CPO  CXC 138 46","2.2.7"},</div><div>      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.5"},</div><div>      {webmachine,"webmachine","1.10.3-rmq3.1.5-gite9359c7"},</div><div>      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.1.5-git680dba8"},</div><div>      {xmerl,"XML parser","1.2.10"},</div><div>      {inets,"INETS  CXC 138 49","5.7.1"},</div><div>      {mnesia,"MNESIA  CXC 138 12","4.5"},</div><div>      {amqp_client,"RabbitMQ AMQP Client","3.1.5"},</div><div>      {sasl,"SASL  CXC 138 11","2.1.10"},</div><div>      {stdlib,"ERTS  CXC 138 10","1.17.5"},</div><div>      {kernel,"ERTS  CXC 138 10","2.14.5"}]},</div><div> {os,{unix,linux}},</div><div> {erlang_version,</div><div>     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:30] [kernel-poll:true]\n"},</div><div> {memory,</div><div>     [{total,2384958696},</div><div>      {connection_procs,243278664},</div><div>      {queue_procs,1295680360},</div><div>      {plugins,2061384},</div><div>      {other_proc,189406552},</div><div>      {mnesia,138736},</div><div>      {mgmt_db,54466288},</div><div>      {msg_index,8958408},</div><div>      {other_ets,37974792},</div><div>      {binary,367954632},</div><div>      {code,17812261},</div><div>      {atom,2285609},</div><div>      {other_system,164941010}]},</div><div> {vm_memory_high_watermark,0.9},</div><div> {vm_memory_limit,15137044070},</div><div> {disk_free_limit,1000000000},</div><div> {disk_free,219615227904},</div><div> {file_descriptors,</div><div>     [{total_limit,262040},</div><div>      {total_used,4426},</div><div>      {sockets_limit,235834},</div><div>      {sockets_used,4422}]},</div><div> {processes,[{limit,1048576},{used,39049}]},</div><div> {run_queue,12},</div><div> {uptime,3590787}]</div><div><br></div><div>Cluster status of node rabbit4@rabbitmq153 ...</div><div>[{nodes,[{disc,[rabbit3@rabbitmq152,rabbit4@rabbitmq153]}]},</div><div> {running_nodes,[rabbit3@rabbitmq152,rabbit4@rabbitmq153]},</div><div> {partitions,[]}]</div><div><br></div><div>Application environment of node rabbit4@rabbitmq153 ...</div><div>[{auth_backends,[rabbit_auth_backend_internal]},</div><div> {auth_mechanisms,['PLAIN','AMQPLAIN']},</div><div> {backing_queue_module,rabbit_variable_queue},</div><div> {cluster_nodes,{[],disc}},</div><div> {cluster_partition_handling,ignore},</div><div> {collect_statistics,fine},</div><div> {collect_statistics_interval,5000},</div><div> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},</div><div> {default_user,<<"guest">>},</div><div> {default_user_tags,[administrator]},</div><div> {default_vhost,<<"/">>},</div><div> {delegate_count,16},</div><div> {disk_free_limit,1000000000},</div><div> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},</div><div> {error_logger,{file,"/data/yy/log/rabbitmq/rabbit4@rabbitmq153.log"}},</div><div> {frame_max,131072},</div><div> {heartbeat,600},</div><div> {hipe_compile,false},</div><div> {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,</div><div>                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,</div><div>                rabbit_basic,rabbit_event,lists,queue,priority_queue,</div><div>                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,</div><div>                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,</div><div>                rabbit_amqqueue_process,rabbit_variable_queue,</div><div>                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,</div><div>                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,</div><div>                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,</div><div>                file_handle_cache,rabbit_msg_store,array,</div><div>                rabbit_msg_store_ets_index,rabbit_msg_file,</div><div>                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,</div><div>                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,</div><div>                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,</div><div>                ssl]},</div><div> {included_applications,[]},</div><div> {log_levels,[{connection,info}]},</div><div> {msg_store_file_size_limit,16777216},</div><div> {msg_store_index_module,rabbit_msg_store_ets_index},</div><div> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins"},</div><div> {plugins_expand_dir,"/data/rabbitmq/rabbit4@rabbitmq153-plugins-expand"},</div><div> {queue_index_max_journal_entries,65536},</div><div> {reverse_dns_lookups,false},</div><div> {sasl_error_logger,{file,"/data/yy/log/rabbitmq/rabbit4@rabbitmq153-sasl.log"}},</div><div> {server_properties,[]},</div><div> {ssl_apps,[asn1,crypto,public_key,ssl]},</div><div> {ssl_cert_login_from,distinguished_name},</div><div> {ssl_listeners,[]},</div><div> {ssl_options,[]},</div><div> {tcp_listen_options,[binary,</div><div>                      {packet,raw},</div><div>                      {reuseaddr,true},</div><div>                      {backlog,128},</div><div>                      {nodelay,true},</div><div>                      {linger,{true,0}},</div><div>                      {exit_on_close,false}]},</div><div> {tcp_listeners,[5672]},</div><div> {trace_vhosts,[]},</div><div> {vm_memory_high_watermark,0.9},</div><div> {vm_memory_high_watermark_paging_ratio,0.9}]</div><div><br></div><div>Status of node rabbit3@rabbitmq152 ...</div><div>[{pid,21118},</div><div> {running_applications,</div><div>     [{rabbitmq_federation_management,"RabbitMQ Federation Management",</div><div>          "3.1.5"},</div><div>      {rabbitmq_management,"RabbitMQ Management Console","3.1.5"},</div><div>      {rabbitmq_federation,"RabbitMQ Federation","3.1.5"},</div><div>      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.5"},</div><div>      {rabbit,"RabbitMQ","3.1.5"},</div><div>      {os_mon,"CPO  CXC 138 46","2.2.7"},</div><div>      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.5"},</div><div>      {webmachine,"webmachine","1.10.3-rmq3.1.5-gite9359c7"},</div><div>      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.1.5-git680dba8"},</div><div>      {xmerl,"XML parser","1.2.10"},</div><div>      {inets,"INETS  CXC 138 49","5.7.1"},</div><div>      {mnesia,"MNESIA  CXC 138 12","4.5"},</div><div>      {amqp_client,"RabbitMQ AMQP Client","3.1.5"},</div><div>      {sasl,"SASL  CXC 138 11","2.1.10"},</div><div>      {stdlib,"ERTS  CXC 138 10","1.17.5"},</div><div>      {kernel,"ERTS  CXC 138 10","2.14.5"}]},</div><div> {os,{unix,linux}},</div><div> {erlang_version,</div><div>     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:30] [kernel-poll:true]\n"},</div><div> {memory,</div><div>     [{total,1948612952},</div><div>      {connection_procs,273728912},</div><div>      {queue_procs,1262564272},</div><div>      {plugins,577536},</div><div>      {other_proc,15237064},</div><div>      {mnesia,138720},</div><div>      {mgmt_db,9904},</div><div>      {msg_index,3223064},</div><div>      {other_ets,27246728},</div><div>      {binary,198736656},</div><div>      {code,17812261},</div><div>      {atom,1624553},</div><div>      {other_system,147713282}]},</div><div> {vm_memory_high_watermark,0.9},</div><div> {vm_memory_limit,15136985088},</div><div> {disk_free_limit,1000000000},</div><div> {disk_free,221611429888},</div><div> {file_descriptors,</div><div>     [{total_limit,262040},</div><div>      {total_used,1010},</div><div>      {sockets_limit,235834},</div><div>      {sockets_used,1006}]},</div><div> {processes,[{limit,1048576},{used,9175}]},</div><div> {run_queue,0},</div><div> {uptime,47734}]</div><div><br></div><div>Cluster status of node rabbit3@rabbitmq152 ...</div><div>[{nodes,[{disc,[rabbit3@rabbitmq152,rabbit4@rabbitmq153]}]},</div><div> {running_nodes,[rabbit4@rabbitmq153,rabbit3@rabbitmq152]},</div><div> {partitions,[]}]</div><div><br></div><div>Application environment of node rabbit3@rabbitmq152 ...</div><div>[{auth_backends,[rabbit_auth_backend_internal]},</div><div> {auth_mechanisms,['PLAIN','AMQPLAIN']},</div><div> {backing_queue_module,rabbit_variable_queue},</div><div> {cluster_nodes,{[],disc}},</div><div> {cluster_partition_handling,ignore},</div><div> {collect_statistics,fine},</div><div> {collect_statistics_interval,5000},</div><div> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},</div><div> {default_user,<<"guest">>},</div><div> {default_user_tags,[administrator]},</div><div> {default_vhost,<<"/">>},</div><div> {delegate_count,16},</div><div> {disk_free_limit,1000000000},</div><div> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},</div><div> {error_logger,{file,"/data/yy/log/rabbitmq/rabbit3@rabbitmq152.log"}},</div><div> {frame_max,131072},</div><div> {heartbeat,600},</div><div> {hipe_compile,false},</div><div> {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,</div><div>                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,</div><div>                rabbit_basic,rabbit_event,lists,queue,priority_queue,</div><div>                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,</div><div>                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,</div><div>                rabbit_amqqueue_process,rabbit_variable_queue,</div><div>                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,</div><div>                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,</div><div>                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,</div><div>                file_handle_cache,rabbit_msg_store,array,</div><div>                rabbit_msg_store_ets_index,rabbit_msg_file,</div><div>                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,</div><div>                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,</div><div>                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,</div><div>                ssl]},</div><div> {included_applications,[]},</div><div> {log_levels,[{connection,info}]},</div><div> {msg_store_file_size_limit,16777216},</div><div> {msg_store_index_module,rabbit_msg_store_ets_index},</div><div> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins"},</div><div> {plugins_expand_dir,"/data/rabbitmq/rabbit3@rabbitmq152-plugins-expand"},</div><div> {queue_index_max_journal_entries,65536},</div><div> {reverse_dns_lookups,false},</div><div> {sasl_error_logger,{file,"/data/yy/log/rabbitmq/rabbit3@rabbitmq152-sasl.log"}},</div><div> {server_properties,[]},</div><div> {ssl_apps,[asn1,crypto,public_key,ssl]},</div><div> {ssl_cert_login_from,distinguished_name},</div><div> {ssl_listeners,[]},</div><div> {ssl_options,[]},</div><div> {tcp_listen_options,[binary,</div><div>                      {packet,raw},</div><div>                      {reuseaddr,true},</div><div>                      {backlog,128},</div><div>                      {nodelay,true},</div><div>                      {linger,{true,0}},</div><div>                      {exit_on_close,false}]},</div><div> {tcp_listeners,[5672]},</div><div> {trace_vhosts,[]},</div><div> {vm_memory_high_watermark,0.9},</div><div> {vm_memory_high_watermark_paging_ratio,0.9}]</div><div><br></div><div>Connections:</div><div>(the report stopped here and no further info show)</div></div><div><br></div></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>