[rabbitmq-discuss] RabbitMQ blocking issue

Konar Peter (Nokia-LC/Chicago) peter.konar at nokia.com
Tue Feb 12 13:27:19 GMT 2013


'rabbitmqctl report
[pkonar at ajtrafp063 ~]$ sudo rabbitmqctl report
Reporting server status on {{2013,2,12},{13,12,55}}

 ...
Status of node rabbit at ajtrafp064 ...
[{pid,3884},
 {running_applications,
     [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.0.2"},
      {rabbitmq_management,"RabbitMQ Management Console","3.0.2"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.0.2"},
      {rabbit,"RabbitMQ","3.0.2"},
      {os_mon,"CPO  CXC 138 46","2.2.7"},
      {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","3.0.2"},
      {webmachine,"webmachine","1.9.1-rmq3.0.2-git52e62bc"},
      {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.0.2-gitd541e9a"},
      {xmerl,"XML parser","1.2.10"},
      {inets,"INETS  CXC 138 49","5.7.1"},
      {mnesia,"MNESIA  CXC 138 12","4.5"},
      {amqp_client,"RabbitMQ AMQP Client","3.0.2"},
      {sasl,"SASL  CXC 138 11","2.1.10"},
      {stdlib,"ERTS  CXC 138 10","1.17.5"},
      {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,35280904},
      {connection_procs,395120},
      {queue_procs,5408},
      {plugins,306296},
      {other_proc,9714296},
      {mnesia,78592},
      {mgmt_db,153144},
      {msg_index,22696},
      {other_ets,1142280},
      {binary,2462008},
      {code,17446642},
      {atom,1553689},
      {other_system,2000733}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6689898496},
 {disk_free_limit,1000000000},
 {disk_free,62383673344},
 {file_descriptors,
     [{total_limit,924},{total_used,7},{sockets_limit,829},{sockets_used,5}]},
 {processes,[{limit,1048576},{used,236}]},
 {run_queue,0},
 {uptime,69961}]

Cluster status of node rabbit at ajtrafp064 ...
[{nodes,[{disc,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]},
 {running_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]},
 {partitions,[]}]

Application environment of node rabbit at ajtrafp064 ...
[{auth_backends,[rabbit_auth_backend_internal]},
 {auth_mechanisms,['PLAIN','AMQPLAIN']},
 {backing_queue_module,rabbit_variable_queue},
 {cluster_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]},
 {collect_statistics,fine},
 {collect_statistics_interval,5000},
 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
 {default_user,<<"guest">>},
 {default_user_tags,[administrator]},
 {default_vhost,<<"/">>},
 {delegate_count,16},
 {disk_free_limit,1000000000},
 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
 {error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp064.log"}},
 {frame_max,131072},
 {heartbeat,600},
 {hipe_compile,false},
 {included_applications,[]},
 {log_levels,[{connection,info}]},
 {msg_store_file_size_limit,16777216},
 {msg_store_index_module,rabbit_msg_store_ets_index},
 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.0.2/sbin/../plugins"},
 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit at ajtrafp064-plugins-expand"},
 {queue_index_max_journal_entries,262144},
 {reverse_dns_lookups,false},
 {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp064-sasl.log"}},
 {server_properties,[]},
 {ssl_cert_login_from,distinguished_name},
 {ssl_listeners,[]},
 {ssl_options,[]},
 {tcp_listen_options,[binary,
                      {packet,raw},
                      {reuseaddr,true},
                      {backlog,128},
                      {nodelay,true},
                      {exit_on_close,false}]},
 {tcp_listeners,[5672]},
 {trace_vhosts,[]},
 {vm_memory_high_watermark,0.4}]

Status of node rabbit at ajtrafp063 ...
[{pid,3873},
 {running_applications,
     [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.0.2"},
      {rabbitmq_management,"RabbitMQ Management Console","3.0.2"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.0.2"},
      {rabbit,"RabbitMQ","3.0.2"},
      {os_mon,"CPO  CXC 138 46","2.2.7"},
      {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","3.0.2"},
      {webmachine,"webmachine","1.9.1-rmq3.0.2-git52e62bc"},
      {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.0.2-gitd541e9a"},
      {xmerl,"XML parser","1.2.10"},
      {inets,"INETS  CXC 138 49","5.7.1"},
      {mnesia,"MNESIA  CXC 138 12","4.5"},
      {amqp_client,"RabbitMQ AMQP Client","3.0.2"},
      {sasl,"SASL  CXC 138 11","2.1.10"},
      {stdlib,"ERTS  CXC 138 10","1.17.5"},
      {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,57276528},
      {connection_procs,134480},
      {queue_procs,21076560},
      {plugins,81960},
      {other_proc,9372096},
      {mnesia,79040},
      {mgmt_db,9904},
      {msg_index,249152},
      {other_ets,2159288},
      {binary,3100872},
      {code,17446642},
      {atom,1553689},
      {other_system,2012845}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6689898496},
 {disk_free_limit,1000000000},
 {disk_free,62375903232},
 {file_descriptors,
     [{total_limit,924},{total_used,7},{sockets_limit,829},{sockets_used,4}]},
 {processes,[{limit,1048576},{used,226}]},
 {run_queue,0},
 {uptime,56951}]

Cluster status of node rabbit at ajtrafp063 ...
[{nodes,[{disc,[rabbit at ajtrafp063,rabbit at ajtrafp064]}]},
 {running_nodes,[rabbit at ajtrafp064,rabbit at ajtrafp063]},
 {partitions,[]}]

Application environment of node rabbit at ajtrafp063 ...
[{auth_backends,[rabbit_auth_backend_internal]},
 {auth_mechanisms,['PLAIN','AMQPLAIN']},
 {backing_queue_module,rabbit_variable_queue},
 {cluster_nodes,[rabbit at ajtrafp063,rabbit at ajtrafp064]},
 {collect_statistics,fine},
 {collect_statistics_interval,5000},
 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
 {default_user,<<"guest">>},
 {default_user_tags,[administrator]},
 {default_vhost,<<"/">>},
 {delegate_count,16},
 {disk_free_limit,1000000000},
 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
 {error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp063.log"}},
 {frame_max,131072},
 {heartbeat,600},
 {hipe_compile,false},
 {included_applications,[]},
 {log_levels,[{connection,info}]},
 {msg_store_file_size_limit,16777216},
 {msg_store_index_module,rabbit_msg_store_ets_index},
 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.0.2/sbin/../plugins"},
 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit at ajtrafp063-plugins-expand"},
 {queue_index_max_journal_entries,262144},
 {reverse_dns_lookups,false},
 {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ajtrafp063-sasl.log"}},
 {server_properties,[]},
 {ssl_cert_login_from,distinguished_name},
 {ssl_listeners,[]},
 {ssl_options,[]},
 {tcp_listen_options,[binary,
                      {packet,raw},
                      {reuseaddr,true},
                      {backlog,128},
                      {nodelay,true},
                      {exit_on_close,false}]},
 {tcp_listeners,[5672]},
 {trace_vhosts,[]},
 {vm_memory_high_watermark,0.4}]

Connections:
pid     name    port    peer_port       host    peer_host       ssl     peer_cert_subject       peer_cert_issuer        peer_cert_validity      auth_mechanism  ssl_protocol    ssl_key_exchange        ssl_cipher      ssl_hash        protocol        user    vhost     timeout frame_max       client_properties       recv_oct        recv_cnt        send_oct        send_cnt        send_pend       state   last_blocked_by last_blocked_age        channels
<rabbit at ajtrafp063.3.3992.0>    10.196.42.119:48485 -> 10.196.42.21:5672        5672    48485   10.196.42.21    10.196.42.119   false                           PLAIN                                   {0,9,1} guest   /       600     131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 149398118       6460971 2122464675      7082940 0       running none    infinity        1
<rabbit at ajtrafp063.3.4254.0>    10.196.42.120:59485 -> 10.196.42.21:5672        5672    59485   10.196.42.21    10.196.42.120   false                           PLAIN                                   {0,9,1} guest   /       600     131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 109362206       4866571 3624114636      5187935 0       running none    infinity        1
<rabbit at ajtrafp063.3.20125.0>   147.243.130.14:34613 -> 10.196.42.21:5672       5672    34613   10.196.42.21    147.243.130.14  false                           PLAIN                                   {0,9,1} guest   /       600     131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 2254516 41540   96969022        93165   0       running none    infinity        1
<rabbit at ajtrafp064.1.14970.129> 10.196.42.182:50012 -> 10.196.42.22:5672        5672    50012   10.196.42.22    10.196.42.182   false                           PLAIN                                   {0,9,1} guest   /       600     131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2013 VMware, Inc."},{"version","3.0.2"}] 892971313       13406695        65127580        4651947 0       running flow    708.153401      2
<rabbit at ajtrafp064.1.13886.130> 10.196.42.180:57261 -> 10.196.42.22:5672        5672    57261   10.196.42.22    10.196.42.180   false                           PLAIN                                   {0,9,1} guest   /       600     131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2013 VMware, Inc."},{"version","3.0.2"}] 991521388       13254940        64474148        4605273 0       running flow    1012.465014     1
<rabbit at ajtrafp064.1.5604.131>  10.196.42.128:38983 -> 10.196.42.22:5672        5672    38983   10.196.42.22    10.196.42.128   false                           PLAIN                                   {0,9,1} guest   /       600     131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2013 VMware, Inc."},{"version","3.0.2"}] 1260443690      13146545        63815924        4558257 0       running flow    713.080137      1
<rabbit at ajtrafp064.1.31824.131> 10.196.42.179:41165 -> 10.196.42.22:5672        5672    41165   10.196.42.22    10.196.42.179   false                           PLAIN                                   {0,9,1} guest   /       600     131072  [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2013 VMware, Inc."},{"version","3.0.2"}] 1207139221      13617335        66191024        4727907 0       running flow    1314.712837     1

Channels:
pid     name    connection      number  user    vhost   transactional   confirm consumer_count  messages_unacknowledged messages_unconfirmed    messages_uncommitted    acks_uncommitted        prefetch_count  client_flow_blocked
<rabbit at ajtrafp063.3.3999.0>    10.196.42.119:48485 -> 10.196.42.21:5672 (1)    <rabbit at ajtrafp063.3.3992.0>    1       guest   /       false   false   1       0       0       0       0       8000    false
<rabbit at ajtrafp063.3.4260.0>    10.196.42.120:59485 -> 10.196.42.21:5672 (1)    <rabbit at ajtrafp063.3.4254.0>    1       guest   /       false   false   1       0       0       0       0       8000    false
<rabbit at ajtrafp063.3.20131.0>   147.243.130.14:34613 -> 10.196.42.21:5672 (1)   <rabbit at ajtrafp063.3.20125.0>   1       guest   /       false   false   1       0       0       0       0       8000    false
<rabbit at ajtrafp064.1.8630.1251> 10.196.42.182:50012 -> 10.196.42.22:5672 (44154)        <rabbit at ajtrafp064.1.14970.129> 44154   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp064.1.8634.1251> 10.196.42.182:50012 -> 10.196.42.22:5672 (44155)        <rabbit at ajtrafp064.1.14970.129> 44155   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp064.1.2184.1262> 10.196.42.128:38983 -> 10.196.42.22:5672 (62655)        <rabbit at ajtrafp064.1.5604.131>  62655   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp064.1.2484.1262> 10.196.42.180:57261 -> 10.196.42.22:5672 (27264)        <rabbit at ajtrafp064.1.13886.130> 27264   guest   /       false   false   0       0       0       0       0       0       false
<rabbit at ajtrafp064.1.2981.1262> 10.196.42.179:41165 -> 10.196.42.22:5672 (7621) <rabbit at ajtrafp064.1.31824.131> 7621    guest   /       false   false   0       0       0       0       0       0       false

Queues on /:
pid     name    durable auto_delete     arguments       owner_pid       policy  exclusive_consumer_pid  exclusive_consumer_tag  messages_ready  messages_unacknowledged messages        consumers       active_consumers        memory  slave_pids      synchronised_slave_pids   backing_queue_status
<rabbit at ajtrafp063.3.626.0>     Something       true    false   [{"x-message-ttl",30000}]                                       0       0       0       0       0       21037632                        [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,72530278}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,1946.202937736505}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
<rabbit at ajtrafp063.3.4000.0>    b2bserverajtrafd015.si.atl01.locn.s.nokia.com   false   true    [{"x-message-ttl",45000}]       <rabbit at ajtrafp063.3.3992.0>                            0       0       0       1       1       23136                   [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,7114159}, {persistent_count,0}, {avg_ingress_rate,429.0251091151943}, {avg_egress_rate,429.0251091151943}, {avg_ack_ingress_rate,429.0251091151943}, {avg_ack_egress_rate,429.16893248398924}]
<rabbit at ajtrafp063.3.4261.0>    b2bserverajtrafd016.si.atl01.locn.s.nokia.com   false   true    [{"x-message-ttl",45000}]       <rabbit at ajtrafp063.3.4254.0>                            0       0       0       1       1       27568                   [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,5207687}, {persistent_count,0}, {avg_ingress_rate,349.00662555488776}, {avg_egress_rate,349.00662555488776}, {avg_ack_ingress_rate,349.00662555488776}, {avg_ack_egress_rate,349.1704784307351}]
<rabbit at ajtrafp063.3.20132.0>   b2bserver-dicksMBPlocal.local   false   true    [{"x-message-ttl",45000}]       <rabbit at ajtrafp063.3.20125.0>                           0       0       0       1       1       23136                   [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,107319}, {persistent_count,0}, {avg_ingress_rate,453.20681423716616}, {avg_egress_rate,453.20681423716616}, {avg_ack_ingress_rate,453.20681423716616}, {avg_ack_egress_rate,461.1071676868251}]

Exchanges on /:
name    type    durable auto_delete     internal        arguments       policy
        direct  true    false   false   []
amq.direct      direct  true    false   false   []
amq.fanout      fanout  true    false   false   []
amq.headers     headers true    false   false   []
amq.match       headers true    false   false   []
amq.rabbitmq.log        topic   true    false   false   []
amq.rabbitmq.trace      topic   true    false   false   []
amq.topic       topic   true    false   false   []
clusterStatus   topic   true    false   false   []
flowData        topic   true    false   false   []
products        topic   true    false   false   []

Bindings on /:
source_name     source_kind     destination_name        destination_kind        routing_key     arguments
        exchange        Something       queue   Something       []
        exchange        b2bserver-dicksMBPlocal.local   queue   b2bserver-dicksMBPlocal.local   []
        exchange        b2bserverajtrafd015.si.atl01.locn.s.nokia.com   queue   b2bserverajtrafd015.si.atl01.locn.s.nokia.com   []
        exchange        b2bserverajtrafd016.si.atl01.locn.s.nokia.com   queue   b2bserverajtrafd016.si.atl01.locn.s.nokia.com   []
flowData        exchange        Something       queue   #       []
flowData        exchange        b2bserver-dicksMBPlocal.local   queue   1_0.107.*       []
flowData        exchange        b2bserver-dicksMBPlocal.local   queue   1_0.D01.*       []
flowData        exchange        b2bserverajtrafd015.si.atl01.locn.s.nokia.com   queue   1_0.D01.*       []
flowData        exchange        b2bserverajtrafd016.si.atl01.locn.s.nokia.com   queue   1_0.F32.*       []

Consumers on /:
queue_name      channel_pid     consumer_tag    ack_required
b2bserver-dicksMBPlocal.local   <rabbit at ajtrafp063.3.20131.0>   amq.ctag-7wgn3n8B8BpDFuw15dCNUQ true
b2bserverajtrafd015.si.atl01.locn.s.nokia.com   <rabbit at ajtrafp063.3.3999.0>    amq.ctag-cGLyM0l72ST1c05mwdn9eQ true
b2bserverajtrafd016.si.atl01.locn.s.nokia.com   <rabbit at ajtrafp063.3.4260.0>    amq.ctag-KBO1GEzW_LGGUBEXFu4Xag true

Permissions on /:
user    configure       write   read
guest   .*      .*      .*

Policies on /:

Parameters on /:

...done.

RabbitMQ Netstat
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 :::5672                     :::*                        LISTEN
tcp        0      0 ::ffff:10.196.42.21:5672    ::ffff:10.196.42.119:48485  ESTABLISHED
tcp        0      0 ::ffff:10.196.42.21:5672    ::ffff:10.196.42.120:59485  ESTABLISHED
tcp        0      0 ::ffff:10.196.42.21:5672    ::ffff:147.243.130.14:34613 ESTABLISHED

TSA Client Netstat
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0 246112 ::ffff:10.196.42.182:50012  ::ffff:10.196.42.22:5672    ESTABLISHED


Jstack:
"pool-6-thread-3" prio=10 tid=0x00007fc76050b800 nid=0x28b2 runnable [0x00007fc4aadeb000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketOutputStream.$$YJP$$socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        - locked <0x000000063abd4638> (a java.io.BufferedOutputStream)
        at java.io.DataOutputStream.flush(DataOutputStream.java:123)
        at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:142)
        at com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:488)
        at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:125)
        at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)
        - locked <0x000000060f49ed88> (a java.lang.Object)
        at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:292)
        - locked <0x000000060f49ed88> (a java.lang.Object)
        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:634)
        at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
        - locked <0x000000060f49f2f8> (a java.lang.Object)
        at $Proxy59.basicPublish(Unknown Source)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:648)
        at org.springframework.amqp.rabbit.core.RabbitTemplate$1.doInRabbit(RabbitTemplate.java:331)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:600)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:329)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:323)
        at com.nokia.traffic.flowdata.transport.rabbitmq.RabbitMQSender.send(RabbitMQSender.java:53)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend_aroundBody4(TrafficStateAccumulatorMP.java:183)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure5.run(TrafficStateAccumulatorMP.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.dispatchOutput(MessageGroupingAspect.java:70)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend(TrafficStateAccumulatorMP.java:182)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.trySendingALinear(TrafficStateAccumulatorMP.java:140)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear_aroundBody2(TrafficStateAccumulatorMP.java:115)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure3.run(TrafficStateAccumulatorMP.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.outputHandler(MessageGroupingAspect.java:231)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear(TrafficStateAccumulatorMP.java:113)
        at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at com.nokia.dempsy.container.internal.LifecycleHelper.invokeOutput(LifecycleHelper.java:178)
        at com.nokia.dempsy.container.MpContainer.invokeOperation(MpContainer.java:1160)
        at com.nokia.dempsy.container.MpContainer.access$600(MpContainer.java:70)
        at com.nokia.dempsy.container.MpContainer$6.run(MpContainer.java:924)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)


Thanks
Peter Konar
________________________________________
From: ext Matthias Radestock [matthias at rabbitmq.com]
Sent: Monday, February 11, 2013 3:58 PM
To: Konar Peter (Nokia-LC/Chicago)
Cc: Carroll James (Nokia-LC/Malvern); Discussions about RabbitMQ; Mccuiston Mitchell (Nokia-LC/Chicago); Lieber Dick (Nokia-LC/Chicago); Johnson Anthony (Nokia-LC/Alpharetta); Yu Jerry.3 (Nokia-LC/Alpharetta)
Subject: Re: [rabbitmq-discuss] RabbitMQ blocking issue

Peter,

On 11/02/13 21:44, Konar Peter (Nokia-LC/Chicago) wrote:
> I have upgraded client and server to 3.0.2 to get heartbeats.

Great, that will make it easier to track down any problem you may see.
More generally though note that heartbeats can be enabled in older
clients; all that changed in 3.0.x is that the server now by default
suggests that heartbeats should be enabled.

> I will get you a "netstat -an" output as soon as we can get this to
> fail.

A snapshot of
- client stacktrace
- server 'rabbitmqctl report' output
- client and server 'netstat -an' output
would be ideal.

Regards,

Matthias.

The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jstack.13:13:52
Type: application/octet-stream
Size: 146933 bytes
Desc: jstack.13:13:52
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130212/d783c71e/attachment.obj>


More information about the rabbitmq-discuss mailing list