[rabbitmq-discuss] RabbitMQ blocking issue
Konar Peter (Nokia-LC/Chicago)
peter.konar at nokia.com
Fri Feb 8 20:05:47 GMT 2013
Jstack trace of the connection that is hung.
"pool-5-thread-2" prio=10 tid=0x00007f90445c0000 nid=0x7a4 runnable [0x00007f8d9eef3000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.$$YJP$$socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x00000006c5e900d8> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:142)
at com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:481)
at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:125)
at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)
- locked <0x00000006b14003f0> (a java.lang.Object)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:292)
- locked <0x00000006b14003f0> (a java.lang.Object)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:601)
at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
- locked <0x00000006b1400b10> (a java.lang.Object)
at $Proxy59.basicPublish(Unknown Source)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:648)
at org.springframework.amqp.rabbit.core.RabbitTemplate$1.doInRabbit(RabbitTemplate.java:331)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:600)
at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:329)
at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:323)
at com.nokia.traffic.flowdata.transport.rabbitmq.RabbitMQSender.send(RabbitMQSender.java:53)
at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend_aroundBody4(TrafficStateAccumulatorMP.java:169)
at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure5.run(TrafficStateAccumulatorMP.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.dispatchOutput(MessageGroupingAspect.java:56)
at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend(TrafficStateAccumulatorMP.java:168)
at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.trySendingALinear(TrafficStateAccumulatorMP.java:126)
at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear_aroundBody2(TrafficStateAccumulatorMP.java:99)
at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure3.run(TrafficStateAccumulatorMP.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.outputHandler(MessageGroupingAspect.java:214)
at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear(TrafficStateAccumulatorMP.java:97)
at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.nokia.dempsy.container.internal.LifecycleHelper.invokeOutput(LifecycleHelper.java:178)
at com.nokia.dempsy.container.MpContainer.invokeOperation(MpContainer.java:1160)
at com.nokia.dempsy.container.MpContainer.access$600(MpContainer.java:70)
at com.nokia.dempsy.container.MpContainer$6.run(MpContainer.java:924)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
[pkonar at ajtrafp063 ~]$ sudo rabbitmqctl report
Reporting server status on {{2013,2,8},{20,5,18}}
Status of node rabbit at ajtrafp064 ...
[{pid,8182},
{running_applications,
[{rabbitmq_management_visualiser,"RabbitMQ Visualiser","2.8.7"},
{rabbitmq_management,"RabbitMQ Management Console","2.8.7"},
{rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.8.7"},
{webmachine,"webmachine","1.9.1-rmq2.8.7-git52e62bc"},
{amqp_client,"RabbitMQ AMQP Client","2.8.7"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","2.8.7"},
{rabbit,"RabbitMQ","2.8.7"},
{os_mon,"CPO CXC 138 46","2.2.7"},
{sasl,"SASL CXC 138 11","2.1.10"},
{mochiweb,"MochiMedia Web Server","2.3.1-rmq2.8.7-gitd541e9a"},
{inets,"INETS CXC 138 49","5.7.1"},
{xmerl,"XML parser","1.2.10"},
{mnesia,"MNESIA CXC 138 12","4.5"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory,
[{total,38201376},
{processes,13480144},
{processes_used,13433112},
{system,24721232},
{atom,1538337},
{atom_used,1524400},
{binary,2739608},
{code,17157476},
{ets,1293432}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,6689898496},
{disk_free_limit,1000000000},
{disk_free,62247305216},
{file_descriptors,
[{total_limit,924},{total_used,6},{sockets_limit,829},{sockets_used,4}]},
{processes,[{limit,1048576},{used,205}]},
{run_queue,0},
{uptime,865082}]
Cluster status of node rabbit at ajtrafp064 ...
[{nodes,[{disc,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]},
{running_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]
Application environment of node rabbit at ajtrafp064 ...
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue},
{cluster_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,1000000000},
{error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp064.log"}},
{frame_max,131072},
{hipe_compile,false},
{included_applications,[]},
{log_levels,[{connection,info}]},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{queue_index_max_journal_entries,262144},
{sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp064-sasl.log"}},
{server_properties,[]},
{ssl_cert_login_from,distinguished_name},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{exit_on_close,false}]},
{tcp_listeners,[5672]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.4}]
Status of node rabbit at ajtrafp063 ...
[{pid,1495},
{running_applications,
[{rabbitmq_management_visualiser,"RabbitMQ Visualiser","2.8.7"},
{rabbitmq_management,"RabbitMQ Management Console","2.8.7"},
{rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.8.7"},
{webmachine,"webmachine","1.9.1-rmq2.8.7-git52e62bc"},
{amqp_client,"RabbitMQ AMQP Client","2.8.7"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","2.8.7"},
{rabbit,"RabbitMQ","2.8.7"},
{os_mon,"CPO CXC 138 46","2.2.7"},
{sasl,"SASL CXC 138 11","2.1.10"},
{mochiweb,"MochiMedia Web Server","2.3.1-rmq2.8.7-gitd541e9a"},
{inets,"INETS CXC 138 49","5.7.1"},
{xmerl,"XML parser","1.2.10"},
{mnesia,"MNESIA CXC 138 12","4.5"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
{memory,
[{total,113292568},
{processes,61427424},
{processes_used,61349488},
{system,51865144},
{atom,1538337},
{atom_used,1523704},
{binary,15952560},
{code,17183056},
{ets,15171968}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,6689898496},
{disk_free_limit,1000000000},
{disk_free,62213640192},
{file_descriptors,
[{total_limit,924},{total_used,11},{sockets_limit,829},{sockets_used,8}]},
{processes,[{limit,1048576},{used,237}]},
{run_queue,0},
{uptime,74168}]
Cluster status of node rabbit at ajtrafp063 ...
[{nodes,[{disc,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]},
{running_nodes,[rabbit at ajtrafp064,rabbit at ajtrafp063]}]
Application environment of node rabbit at ajtrafp063 ...
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue},
{cluster_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,1000000000},
{error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp063.log"}},
{frame_max,131072},
{hipe_compile,false},
{included_applications,[]},
{log_levels,[{connection,info}]},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{queue_index_max_journal_entries,262144},
{sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp063-sasl.log"}},
{server_properties,[]},
{ssl_cert_login_from,distinguished_name},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{exit_on_close,false}]},
{tcp_listeners,[5672]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.4}]
Connections:
pid name address port peer_address peer_port ssl peer_cert_subject peer_cert_issuer peer_cert_validity auth_mechanism ssl_protocol ssl_key_exchange ssl_cipher ssl_hash protocol user vhost timeout frame_max client_properties recv_oct recv_cnt send_oct send_cnt send_pend state last_blocked_by last_blocked_age channels
<rabbit at ajtrafp064.3.17615.445> 10.196.42.119:37257 -> 10.196.42.22:5672 10.196.42.22 5672 10.196.42.119 37257 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 211368165 9198234 483965816 10062679 0 running none infinity 2
<rabbit at ajtrafp064.3.17602.445> 10.196.42.120:54628 -> 10.196.42.22:5672 10.196.42.22 5672 10.196.42.120 54628 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 155539518 6925521 856651504 7405560 0 running none infinity 2
<rabbit at ajtrafp064.3.17674.445> 10.196.42.182:34772 -> 10.196.42.22:5672 10.196.42.22 5672 10.196.42.182 34772 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 8982892 10258 59980 4261 0 running none infinity 1
<rabbit at ajtrafp063.1.17262.888> 10.196.42.181:44585 -> 10.196.42.21:5672 10.196.42.21 5672 10.196.42.181 44585 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 3042287025 1833703 1623444 115937 0 running flow 2051.757231 1
<rabbit at ajtrafp063.1.7689.860> 10.196.42.182:52070 -> 10.196.42.21:5672 10.196.42.21 5672 10.196.42.182 52070 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 3569900672 2105694 1887892 134826 0 running flow 2712.356091 2
<rabbit at ajtrafp063.1.412.0> 10.196.42.128:54376 -> 10.196.42.21:5672 10.196.42.21 5672 10.196.42.128 54376 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 19086 33 480 11 0 running none infinity 1
<rabbit at ajtrafp063.1.5685.336> 10.196.42.180:34088 -> 10.196.42.21:5672 10.196.42.21 5672 10.196.42.180 34088 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 4074977814 6637192 50504276 3607425 0 running flow 3855.880187 1
<rabbit at ajtrafp063.1.23419.903> 10.196.129.114:47978 -> 10.196.42.21:5672 10.196.42.21 5672 10.196.129.114 47978 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 262583397 2920737 939686596 7134486 0 running none infinity 1
<rabbit at ajtrafp063.1.6391.783> 10.196.42.128:58015 -> 10.196.42.21:5672 10.196.42.21 5672 10.196.42.128 58015 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 678123470 1736589 15442188 1102990 0 running flow 3795.312193 2
<rabbit at ajtrafp063.1.457.0> 10.196.42.180:42353 -> 10.196.42.21:5672 10.196.42.21 5672 10.196.42.180 42353 false PLAIN {0,9,1} guest / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 25881171 35488 88596 6305 0 running none infinity 1
Channels:
pid name connection number user vhost transactional confirm consumer_count messages_unacknowledged messages_unconfirmed messages_uncommitted acks_uncommitted prefetch_count client_flow_blocked
<rabbit at ajtrafp064.3.17606.445> 10.196.42.120:54628 -> 10.196.42.22:5672 (1) <rabbit at ajtrafp064.3.17602.445> 1 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp064.3.17611.445> 10.196.42.120:54628 -> 10.196.42.22:5672 (2) <rabbit at ajtrafp064.3.17602.445> 2 guest / false false 1 0 0 0 0 8000 false
<rabbit at ajtrafp064.3.17619.445> 10.196.42.119:37257 -> 10.196.42.22:5672 (1) <rabbit at ajtrafp064.3.17615.445> 1 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp064.3.17624.445> 10.196.42.119:37257 -> 10.196.42.22:5672 (2) <rabbit at ajtrafp064.3.17615.445> 2 guest / false false 1 0 0 0 0 8000 false
<rabbit at ajtrafp064.3.27234.445> 10.196.42.182:34772 -> 10.196.42.22:5672 (1710) <rabbit at ajtrafp064.3.17674.445> 1710 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.423.0> 10.196.42.128:54376 -> 10.196.42.21:5672 (2) <rabbit at ajtrafp063.1.412.0> 2 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.13080.0> 10.196.42.180:42353 -> 10.196.42.21:5672 (798) <rabbit at ajtrafp063.1.457.0> 798 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.23423.903> 10.196.129.114:47978 -> 10.196.42.21:5672 (1) <rabbit at ajtrafp063.1.23419.903> 1 guest / false false 1 0 0 0 0 8000 false
<rabbit at ajtrafp063.1.7295.958> 10.196.42.128:58015 -> 10.196.42.21:5672 (55418) <rabbit at ajtrafp063.1.6391.783> 55418 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.7318.958> 10.196.42.128:58015 -> 10.196.42.21:5672 (55419) <rabbit at ajtrafp063.1.6391.783> 55419 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.14305.964> 10.196.42.182:52070 -> 10.196.42.21:5672 (26027) <rabbit at ajtrafp063.1.7689.860> 26027 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.14309.964> 10.196.42.182:52070 -> 10.196.42.21:5672 (26028) <rabbit at ajtrafp063.1.7689.860> 26028 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.25715.968> 10.196.42.181:44585 -> 10.196.42.21:5672 (22148) <rabbit at ajtrafp063.1.17262.888> 22148 guest / false false 0 0 0 0 0 0 false
<rabbit at ajtrafp063.1.18018.977> 10.196.42.180:34088 -> 10.196.42.21:5672 (41340) <rabbit at ajtrafp063.1.5685.336> 41340 guest / false false 0 0 0 0 0 0 false
Queues on /:
pid name durable auto_delete arguments owner_pid slave_pids synchronised_slave_pids exclusive_consumer_pid exclusive_consumer_tag messages_ready messages_unacknowledged messages consumers memory backing_queue_status
<rabbit at ajtrafp063.1.23424.903> b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com false true [{"x-message-ttl",45000}] <rabbit at ajtrafp063.1.23419.903> 0 0 0 1 22976 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,12503849}, {persistent_count,0}, {avg_ingress_rate,1249.1263854803442}, {avg_egress_rate,1249.1263854803442}, {avg_ack_ingress_rate,1249.1263854803442}, {avg_ack_egress_rate,1249.1263854803442}]
<rabbit at ajtrafp064.3.17607.445> b2bserverajtrafd016.si.atl01.locn.s.nokia.com false true [{"x-message-ttl",45000}] <rabbit at ajtrafp064.3.17602.445> 0 0 0 1 23520 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,7406606}, {persistent_count,0}, {avg_ingress_rate,111.79187855499954}, {avg_egress_rate,111.79187855499954}, {avg_ack_ingress_rate,111.79187855499954}, {avg_ack_egress_rate,111.79187855499954}]
<rabbit at ajtrafp063.1.259.0> Something true false [{"x-message-ttl",30000}] 19262 0 19262 0 47743560 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,19262}, {len,19262}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,19262}, {ram_ack_count,0}, {next_seq_id,953725980}, {persistent_count,19262}, {avg_ingress_rate,934.2094154568135}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
<rabbit at ajtrafp064.3.17620.445> b2bserverajtrafd015.si.atl01.locn.s.nokia.com false true [{"x-message-ttl",45000}] <rabbit at ajtrafp064.3.17615.445> 0 0 0 1 22912 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,10065113}, {persistent_count,0}, {avg_ingress_rate,154.37923364290802}, {avg_egress_rate,154.37923364290802}, {avg_ack_ingress_rate,154.37923364290802}, {avg_ack_egress_rate,154.37923364290802}]
Exchanges on /:
name type durable auto_delete internal arguments
amq.direct direct true false false []
products topic true false false []
amq.topic topic true false false []
clusterStatus topic true false false []
amq.rabbitmq.trace topic true false false []
flowData topic true false false []
amq.rabbitmq.log topic true false false []
amq.fanout fanout true false false []
amq.headers headers true false false []
direct true false false []
amq.match headers true false false []
Bindings on /:
source_name source_kind destination_name destination_kind routing_key arguments
exchange Something queue Something []
exchange b2bserverajtrafd015.si.atl01.locn.s.nokia.com queue b2bserverajtrafd015.si.atl01.locn.s.nokia.com []
exchange b2bserverajtrafd016.si.atl01.locn.s.nokia.com queue b2bserverajtrafd016.si.atl01.locn.s.nokia.com []
exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com []
flowData exchange Something queue # []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.018.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.105.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.106.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.107.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.242.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.305.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.336.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.409.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.501.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.601.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.617.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.701.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.817.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.842.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.909.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.A01.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.C07.* []
flowData exchange b2bserverajtrafd015.si.atl01.locn.s.nokia.com queue 1_0.D01.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.D01.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.E17.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.E33.* []
flowData exchange b2bserverajtrafd016.si.atl01.locn.s.nokia.com queue 1_0.F32.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.F32.* []
flowData exchange b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com queue 1_0.F49.* []
Consumers on /:
queue_name channel_pid consumer_tag ack_required
b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com <rabbit at ajtrafp063.1.23423.903> amq.ctag-QAPE8DIK-g54ofPr2ioHtj true
b2bserverajtrafd016.si.atl01.locn.s.nokia.com <rabbit at ajtrafp064.3.17611.445> amq.ctag-gQ8mSRj6kILAjSl9L6cLOE true
b2bserverajtrafd015.si.atl01.locn.s.nokia.com <rabbit at ajtrafp064.3.17624.445> amq.ctag-AKCYDlYI6S0jhWNsHp9zu8 true
Permissions on /:
user configure write read
guest .* .* .*
End of server status report
...done.
[pkonar at ajtrafp063 ~]$
Thanks
Peter Konar
________________________________________
From: ext Matthias Radestock [matthias at rabbitmq.com]
Sent: Friday, February 08, 2013 12:13 PM
To: Carroll James (Nokia-LC/Malvern)
Cc: Konar Peter (Nokia-LC/Chicago); Discussions about RabbitMQ; Mccuiston Mitchell (Nokia-LC/Chicago); Lieber Dick (Nokia-LC/Chicago); Johnson Anthony (Nokia-LC/Alpharetta); Yu Jerry.3 (Nokia-LC/Alpharetta)
Subject: Re: [rabbitmq-discuss] RabbitMQ blocking issue
On 08/02/13 17:59, Carroll James (Nokia-LC/Malvern) wrote:
> As far as where it's hanging on the client side: I don't have the
> exact rabbit call stack (I'll save off the jstack next time) but it's
> inside the amqp client code and inside of the native socketWrite
> call. Both the send buffer on the client side and the recv buffer on
> the server side are full. While the server is still maintaining the
> connection it's not reading from the socket. Since the client is
> written in terms of blocking java.io, the client hangs waiting for
> the server to read the data.
Hmmm, in which case please post the output of 'rabbitmqctl report' as well.
Peter posted some 'rabbitmqctl list_connections' output but
unfortunately that did not include the 'state' field so it's impossible
to tell whether the connections were being throttled at the time.
Matthias.
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
More information about the rabbitmq-discuss
mailing list