[rabbitmq-discuss] RabbitMQ blocking issue

Konar Peter (Nokia-LC/Chicago) peter.konar at nokia.com
Fri Feb 8 20:05:47 GMT 2013


Jstack trace of the connection that is hung.

"pool-5-thread-2" prio=10 tid=0x00007f90445c0000 nid=0x7a4 runnable [0x00007f8d9eef3000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketOutputStream.$$YJP$$socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        - locked <0x00000006c5e900d8> (a java.io.BufferedOutputStream)
        at java.io.DataOutputStream.flush(DataOutputStream.java:123)
        at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:142)
        at com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:481)
        at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:125)
        at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)
        - locked <0x00000006b14003f0> (a java.lang.Object)
        at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:292)
        - locked <0x00000006b14003f0> (a java.lang.Object)
        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:601)
        at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
        - locked <0x00000006b1400b10> (a java.lang.Object)
        at $Proxy59.basicPublish(Unknown Source)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:648)
        at org.springframework.amqp.rabbit.core.RabbitTemplate$1.doInRabbit(RabbitTemplate.java:331)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:600)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:329)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:323)
        at com.nokia.traffic.flowdata.transport.rabbitmq.RabbitMQSender.send(RabbitMQSender.java:53)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend_aroundBody4(TrafficStateAccumulatorMP.java:169)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure5.run(TrafficStateAccumulatorMP.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.dispatchOutput(MessageGroupingAspect.java:56)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend(TrafficStateAccumulatorMP.java:168)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.trySendingALinear(TrafficStateAccumulatorMP.java:126)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear_aroundBody2(TrafficStateAccumulatorMP.java:99)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure3.run(TrafficStateAccumulatorMP.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.outputHandler(MessageGroupingAspect.java:214)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear(TrafficStateAccumulatorMP.java:97)
        at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at com.nokia.dempsy.container.internal.LifecycleHelper.invokeOutput(LifecycleHelper.java:178)
        at com.nokia.dempsy.container.MpContainer.invokeOperation(MpContainer.java:1160)
        at com.nokia.dempsy.container.MpContainer.access$600(MpContainer.java:70)
        at com.nokia.dempsy.container.MpContainer$6.run(MpContainer.java:924)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)



[pkonar at ajtrafp063 ~]$ sudo rabbitmqctl report
Reporting server status on {{2013,2,8},{20,5,18}}

Status of node rabbit at ajtrafp064 ...
[{pid,8182},
 {running_applications,
     [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","2.8.7"},
      {rabbitmq_management,"RabbitMQ Management Console","2.8.7"},
      {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.8.7"},
      {webmachine,"webmachine","1.9.1-rmq2.8.7-git52e62bc"},
      {amqp_client,"RabbitMQ AMQP Client","2.8.7"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","2.8.7"},
      {rabbit,"RabbitMQ","2.8.7"},
      {os_mon,"CPO  CXC 138 46","2.2.7"},
      {sasl,"SASL  CXC 138 11","2.1.10"},
      {mochiweb,"MochiMedia Web Server","2.3.1-rmq2.8.7-gitd541e9a"},
      {inets,"INETS  CXC 138 49","5.7.1"},
      {xmerl,"XML parser","1.2.10"},
      {mnesia,"MNESIA  CXC 138 12","4.5"},
      {stdlib,"ERTS  CXC 138 10","1.17.5"},
      {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,38201376},
      {processes,13480144},
      {processes_used,13433112},
      {system,24721232},
      {atom,1538337},
      {atom_used,1524400},
      {binary,2739608},
      {code,17157476},
      {ets,1293432}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6689898496},
 {disk_free_limit,1000000000},
 {disk_free,62247305216},
 {file_descriptors,
     [{total_limit,924},{total_used,6},{sockets_limit,829},{sockets_used,4}]},
 {processes,[{limit,1048576},{used,205}]},
 {run_queue,0},
 {uptime,865082}]

Cluster status of node rabbit at ajtrafp064 ...
[{nodes,[{disc,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]},
 {running_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]

Application environment of node rabbit at ajtrafp064 ...
[{auth_backends,[rabbit_auth_backend_internal]},
 {auth_mechanisms,['PLAIN','AMQPLAIN']},
 {backing_queue_module,rabbit_variable_queue},
 {cluster_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]},
 {collect_statistics,fine},
 {collect_statistics_interval,5000},
 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
 {default_user,<<"guest">>},
 {default_user_tags,[administrator]},
 {default_vhost,<<"/">>},
 {delegate_count,16},
 {disk_free_limit,1000000000},
 {error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp064.log"}},
 {frame_max,131072},
 {hipe_compile,false},
 {included_applications,[]},
 {log_levels,[{connection,info}]},
 {msg_store_file_size_limit,16777216},
 {msg_store_index_module,rabbit_msg_store_ets_index},
 {queue_index_max_journal_entries,262144},
 {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp064-sasl.log"}},
 {server_properties,[]},
 {ssl_cert_login_from,distinguished_name},
 {ssl_listeners,[]},
 {ssl_options,[]},
 {tcp_listen_options,[binary,
                      {packet,raw},
                      {reuseaddr,true},
                      {backlog,128},
                      {nodelay,true},
                      {exit_on_close,false}]},
 {tcp_listeners,[5672]},
 {trace_vhosts,[]},
 {vm_memory_high_watermark,0.4}]

Status of node rabbit at ajtrafp063 ...
[{pid,1495},
 {running_applications,
     [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","2.8.7"},
      {rabbitmq_management,"RabbitMQ Management Console","2.8.7"},
      {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.8.7"},
      {webmachine,"webmachine","1.9.1-rmq2.8.7-git52e62bc"},
      {amqp_client,"RabbitMQ AMQP Client","2.8.7"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","2.8.7"},
      {rabbit,"RabbitMQ","2.8.7"},
      {os_mon,"CPO  CXC 138 46","2.2.7"},
      {sasl,"SASL  CXC 138 11","2.1.10"},
      {mochiweb,"MochiMedia Web Server","2.3.1-rmq2.8.7-gitd541e9a"},
      {inets,"INETS  CXC 138 49","5.7.1"},
      {xmerl,"XML parser","1.2.10"},
      {mnesia,"MNESIA  CXC 138 12","4.5"},
      {stdlib,"ERTS  CXC 138 10","1.17.5"},
      {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,113292568},
      {processes,61427424},
      {processes_used,61349488},
      {system,51865144},
      {atom,1538337},
      {atom_used,1523704},
      {binary,15952560},
      {code,17183056},
      {ets,15171968}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6689898496},
 {disk_free_limit,1000000000},
 {disk_free,62213640192},
 {file_descriptors,
     [{total_limit,924},{total_used,11},{sockets_limit,829},{sockets_used,8}]},
 {processes,[{limit,1048576},{used,237}]},
 {run_queue,0},
 {uptime,74168}]

Cluster status of node rabbit at ajtrafp063 ...
[{nodes,[{disc,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]},
 {running_nodes,[rabbit at ajtrafp064,rabbit at ajtrafp063]}]

Application environment of node rabbit at ajtrafp063 ...
[{auth_backends,[rabbit_auth_backend_internal]},
 {auth_mechanisms,['PLAIN','AMQPLAIN']},
 {backing_queue_module,rabbit_variable_queue},
 {cluster_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]},
 {collect_statistics,fine},
 {collect_statistics_interval,5000},
 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
 {default_user,<<"guest">>},
 {default_user_tags,[administrator]},
 {default_vhost,<<"/">>},
 {delegate_count,16},
 {disk_free_limit,1000000000},
 {error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp063.log"}},
 {frame_max,131072},
 {hipe_compile,false},
 {included_applications,[]},
 {log_levels,[{connection,info}]},
 {msg_store_file_size_limit,16777216},
 {msg_store_index_module,rabbit_msg_store_ets_index},
 {queue_index_max_journal_entries,262144},
 {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp063-sasl.log"}},
 {server_properties,[]},
 {ssl_cert_login_from,distinguished_name},
 {ssl_listeners,[]},
 {ssl_options,[]},
 {tcp_listen_options,[binary,
                      {packet,raw},
                      {reuseaddr,true},
                      {backlog,128},
                      {nodelay,true},
                      {exit_on_close,false}]},
 {tcp_listeners,[5672]},
 {trace_vhosts,[]},
 {vm_memory_high_watermark,0.4}]

Connections:
pid     name    address port    peer_address    peer_port       ssl     peer_cert_subject       peer_cert_issuer        peer_cert_validity      auth_mechanism  ssl_protocol    ssl_key_exchange        ssl_cipher      ssl_hash        protocol        user    vhost     timeout frame_max       client_properties       recv_oct        recv_cnt        send_oct        send_cnt        send_pend       state   last_blocked_by last_blocked_age        channels
<rabbit at ajtrafp064.3.17615.445> 10.196.42.119:37257 -> 10.196.42.22:5672        10.196.42.22    5672    10.196.42.119   37257   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 211368165       9198234 483965816       10062679        0       running none    infinity        2
<rabbit at ajtrafp064.3.17602.445> 10.196.42.120:54628 -> 10.196.42.22:5672        10.196.42.22    5672    10.196.42.120   54628   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 155539518       6925521 856651504       7405560 0       running none    infinity        2
<rabbit at ajtrafp064.3.17674.445> 10.196.42.182:34772 -> 10.196.42.22:5672        10.196.42.22    5672    10.196.42.182   34772   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 8982892 10258   59980   4261    0       running none    infinity        1
<rabbit at ajtrafp063.1.17262.888> 10.196.42.181:44585 -> 10.196.42.21:5672        10.196.42.21    5672    10.196.42.181   44585   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 3042287025      1833703 1623444 115937  0       running flow    2051.757231     1
<rabbit at ajtrafp063.1.7689.860>  10.196.42.182:52070 -> 10.196.42.21:5672        10.196.42.21    5672    10.196.42.182   52070   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 3569900672      2105694 1887892 134826  0       running flow    2712.356091     2
<rabbit at ajtrafp063.1.412.0>     10.196.42.128:54376 -> 10.196.42.21:5672        10.196.42.21    5672    10.196.42.128   54376   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 19086   33      480     11      0       running none    infinity        1
<rabbit at ajtrafp063.1.5685.336>  10.196.42.180:34088 -> 10.196.42.21:5672        10.196.42.21    5672    10.196.42.180   34088   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 4074977814      6637192 50504276        3607425 0       running flow    3855.880187     1
<rabbit at ajtrafp063.1.23419.903> 10.196.129.114:47978 -> 10.196.42.21:5672       10.196.42.21    5672    10.196.129.114  47978   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 262583397       2920737 939686596       7134486 0       running none    infinity        1
<rabbit at ajtrafp063.1.6391.783>  10.196.42.128:58015 -> 10.196.42.21:5672        10.196.42.21    5672    10.196.42.128   58015   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 678123470       1736589 15442188        1102990 0       running flow    3795.312193     2
<rabbit at ajtrafp063.1.457.0>     10.196.42.180:42353 -> 10.196.42.21:5672        10.196.42.21    5672    10.196.42.180   42353   false                           PLAIN                                   {0,9,1} guest   /       0       131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.7"}] 25881171        35488   88596   6305    0       running none    infinity        1

Channels:
pid     name    connection      number  user    vhost   transactional   confirm consumer_count  messages_unacknowledged messages_unconfirmed    messages_uncommitted    acks_uncommitted        prefetch_count  client_flow_blocked
<rabbit at ajtrafp064.3.17606.445> 10.196.42.120:54628 -> 10.196.42.22:5672 (1)    <rabbit at ajtrafp064.3.17602.445> 1       guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp064.3.17611.445> 10.196.42.120:54628 -> 10.196.42.22:5672 (2)    <rabbit at ajtrafp064.3.17602.445> 2       guest   /       false   false   1       0       0       0       0       8000    false
<rabbit at ajtrafp064.3.17619.445> 10.196.42.119:37257 -> 10.196.42.22:5672 (1)    <rabbit at ajtrafp064.3.17615.445> 1       guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp064.3.17624.445> 10.196.42.119:37257 -> 10.196.42.22:5672 (2)    <rabbit at ajtrafp064.3.17615.445> 2       guest   /       false   false   1       0       0       0       0       8000    false
<rabbit at ajtrafp064.3.27234.445> 10.196.42.182:34772 -> 10.196.42.22:5672 (1710) <rabbit at ajtrafp064.3.17674.445> 1710    guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.423.0>     10.196.42.128:54376 -> 10.196.42.21:5672 (2)    <rabbit at ajtrafp063.1.412.0>     2       guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.13080.0>   10.196.42.180:42353 -> 10.196.42.21:5672 (798)  <rabbit at ajtrafp063.1.457.0>     798     guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.23423.903> 10.196.129.114:47978 -> 10.196.42.21:5672 (1)   <rabbit at ajtrafp063.1.23419.903> 1       guest   /       false   false   1       0       0       0       0       8000    false
<rabbit at ajtrafp063.1.7295.958>  10.196.42.128:58015 -> 10.196.42.21:5672 (55418)        <rabbit at ajtrafp063.1.6391.783>  55418   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.7318.958>  10.196.42.128:58015 -> 10.196.42.21:5672 (55419)        <rabbit at ajtrafp063.1.6391.783>  55419   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.14305.964> 10.196.42.182:52070 -> 10.196.42.21:5672 (26027)        <rabbit at ajtrafp063.1.7689.860>  26027   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.14309.964> 10.196.42.182:52070 -> 10.196.42.21:5672 (26028)        <rabbit at ajtrafp063.1.7689.860>  26028   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.25715.968> 10.196.42.181:44585 -> 10.196.42.21:5672 (22148)        <rabbit at ajtrafp063.1.17262.888> 22148   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp063.1.18018.977> 10.196.42.180:34088 -> 10.196.42.21:5672 (41340)        <rabbit at ajtrafp063.1.5685.336>  41340   guest   /       false   false   0       0       0       0       0       0       false

Queues on /:
pid     name    durable auto_delete     arguments       owner_pid       slave_pids      synchronised_slave_pids exclusive_consumer_pid  exclusive_consumer_tag  messages_ready  messages_unacknowledged messages        consumers       memory  backing_queue_status
<rabbit at ajtrafp063.1.23424.903> b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  false   true    [{"x-message-ttl",45000}]       <rabbit at ajtrafp063.1.23419.903>                                 0       0       0       1       22976   [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,12503849}, {persistent_count,0}, {avg_ingress_rate,1249.1263854803442}, {avg_egress_rate,1249.1263854803442}, {avg_ack_ingress_rate,1249.1263854803442}, {avg_ack_egress_rate,1249.1263854803442}]
<rabbit at ajtrafp064.3.17607.445> b2bserverajtrafd016.si.atl01.locn.s.nokia.com   false   true    [{"x-message-ttl",45000}]       <rabbit at ajtrafp064.3.17602.445>                                 0       0       0       1       23520   [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,7406606}, {persistent_count,0}, {avg_ingress_rate,111.79187855499954}, {avg_egress_rate,111.79187855499954}, {avg_ack_ingress_rate,111.79187855499954}, {avg_ack_egress_rate,111.79187855499954}]
<rabbit at ajtrafp063.1.259.0>     Something       true    false   [{"x-message-ttl",30000}]                                               19262   0       19262   0       47743560        [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,19262}, {len,19262}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,19262}, {ram_ack_count,0}, {next_seq_id,953725980}, {persistent_count,19262}, {avg_ingress_rate,934.2094154568135}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
<rabbit at ajtrafp064.3.17620.445> b2bserverajtrafd015.si.atl01.locn.s.nokia.com   false   true    [{"x-message-ttl",45000}]       <rabbit at ajtrafp064.3.17615.445>                                 0       0       0       1       22912   [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,10065113}, {persistent_count,0}, {avg_ingress_rate,154.37923364290802}, {avg_egress_rate,154.37923364290802}, {avg_ack_ingress_rate,154.37923364290802}, {avg_ack_egress_rate,154.37923364290802}]

Exchanges on /:
name    type    durable auto_delete     internal        arguments
amq.direct      direct  true    false   false   []
products        topic   true    false   false   []
amq.topic       topic   true    false   false   []
clusterStatus   topic   true    false   false   []
amq.rabbitmq.trace      topic   true    false   false   []
flowData        topic   true    false   false   []
amq.rabbitmq.log        topic   true    false   false   []
amq.fanout      fanout  true    false   false   []
amq.headers     headers true    false   false   []
        direct  true    false   false   []
amq.match       headers true    false   false   []

Bindings on /:
source_name     source_kind     destination_name        destination_kind        routing_key     arguments
        exchange        Something       queue   Something       []
        exchange        b2bserverajtrafd015.si.atl01.locn.s.nokia.com   queue   b2bserverajtrafd015.si.atl01.locn.s.nokia.com   []
        exchange        b2bserverajtrafd016.si.atl01.locn.s.nokia.com   queue   b2bserverajtrafd016.si.atl01.locn.s.nokia.com   []
        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  []
flowData        exchange        Something       queue   #       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.018.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.105.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.106.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.107.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.242.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.305.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.336.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.409.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.501.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.601.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.617.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.701.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.817.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.842.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.909.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.A01.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.C07.*       []
flowData        exchange        b2bserverajtrafd015.si.atl01.locn.s.nokia.com   queue   1_0.D01.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.D01.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.E17.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.E33.*       []
flowData        exchange        b2bserverajtrafd016.si.atl01.locn.s.nokia.com   queue   1_0.F32.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.F32.*       []
flowData        exchange        b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  queue   1_0.F49.*       []

Consumers on /:
queue_name      channel_pid     consumer_tag    ack_required
b2bserveratlvmtrafd008.dv.atl.locn.s.nokia.com  <rabbit at ajtrafp063.1.23423.903> amq.ctag-QAPE8DIK-g54ofPr2ioHtj true
b2bserverajtrafd016.si.atl01.locn.s.nokia.com   <rabbit at ajtrafp064.3.17611.445> amq.ctag-gQ8mSRj6kILAjSl9L6cLOE true
b2bserverajtrafd015.si.atl01.locn.s.nokia.com   <rabbit at ajtrafp064.3.17624.445> amq.ctag-AKCYDlYI6S0jhWNsHp9zu8 true

Permissions on /:
user    configure       write   read
guest   .*      .*      .*

End of server status report
...done.
[pkonar at ajtrafp063 ~]$


Thanks
Peter Konar
________________________________________
From: ext Matthias Radestock [matthias at rabbitmq.com]
Sent: Friday, February 08, 2013 12:13 PM
To: Carroll James (Nokia-LC/Malvern)
Cc: Konar Peter (Nokia-LC/Chicago); Discussions about RabbitMQ; Mccuiston Mitchell (Nokia-LC/Chicago); Lieber Dick (Nokia-LC/Chicago); Johnson Anthony (Nokia-LC/Alpharetta); Yu Jerry.3 (Nokia-LC/Alpharetta)
Subject: Re: [rabbitmq-discuss] RabbitMQ blocking issue

On 08/02/13 17:59, Carroll James (Nokia-LC/Malvern) wrote:
> As far as where it's hanging on the client side: I don't have the
> exact rabbit call stack (I'll save off the jstack next time) but it's
> inside the amqp client code and inside of the native socketWrite
> call. Both the send buffer on the client side and the recv buffer on
> the server side are full. While the server is still maintaining the
> connection it's not reading from the socket. Since the client is
> written in terms of blocking java.io, the client hangs waiting for
> the server to read the data.

Hmmm, in which case please post the output of 'rabbitmqctl report' as well.

Peter posted some 'rabbitmqctl list_connections' output but
unfortunately that did not include the 'state' field so it's impossible
to tell whether the connections were being throttled at the time.

Matthias.

The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.


More information about the rabbitmq-discuss mailing list