[rabbitmq-discuss] RabbitMQ blocking issue

Wed Feb 13 21:27:17 GMT 2013

Happened again

Netstat client

[pkonar at ajtrafp057 ~]$ netstat -an | grep 5672
tcp        0 189824 ::ffff:10.196.42.182:49191  ::ffff:10.196.42.21:5672    ESTABLISHED

netstat server
[pkonar at ajtrafp063 ~]$ netstat -an | grep 182
tcp   574331      0 ::ffff:10.196.42.21:5672    ::ffff:10.196.42.182:49191  ESTABLISHED

"pool-6-thread-3" prio=10 tid=0x00000000018c2000 nid=0x4ed runnable [0x00007f19d34f3000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketOutputStream.$$YJP$$socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        - locked <0x00000007018815c0> (a java.io.BufferedOutputStream)
        at java.io.DataOutputStream.flush(DataOutputStream.java:123)
        at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:142)
        at com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:488)
        at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:125)
        at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)
        - locked <0x00000007512100b8> (a java.lang.Object)
        at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:292)
        - locked <0x00000007512100b8> (a java.lang.Object)
        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:634)
        at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
        - locked <0x00000007512102f8> (a java.lang.Object)
        at $Proxy59.basicPublish(Unknown Source)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:648)
        at org.springframework.amqp.rabbit.core.RabbitTemplate$1.doInRabbit(RabbitTemplate.java:331)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:600)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:329)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:323)
        at com.nokia.traffic.flowdata.transport.rabbitmq.RabbitMQSender.send(RabbitMQSender.java:53)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend_aroundBody4(TrafficStateAccumulatorMP.java:183)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure5.run(TrafficStateAccumulatorMP.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.dispatchOutput(MessageGroupingAspect.java:70)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.doSend(TrafficStateAccumulatorMP.java:182)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.trySendingALinear(TrafficStateAccumulatorMP.java:140)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear_aroundBody2(TrafficStateAccumulatorMP.java:115)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP$AjcClosure3.run(TrafficStateAccumulatorMP.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.nokia.traffic.ttbhm.audit.MessageGroupingAspect.outputHandler(MessageGroupingAspect.java:231)
        at com.nokia.traffic.ttbhm.mp.linearstate.TrafficStateAccumulatorMP.onSendLinear(TrafficStateAccumulatorMP.java:113)
        at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at com.nokia.dempsy.container.internal.LifecycleHelper.invokeOutput(LifecycleHelper.java:178)
        at com.nokia.dempsy.container.MpContainer.invokeOperation(MpContainer.java:1160)
        at com.nokia.dempsy.container.MpContainer.access$600(MpContainer.java:70)
        at com.nokia.dempsy.container.MpContainer$6.run(MpContainer.java:924)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

________________________________________
From: ext Matthias Radestock [matthias at rabbitmq.com]
Sent: Tuesday, February 12, 2013 2:05 PM
To: Discussions about RabbitMQ
Cc: Konar Peter (Nokia-LC/Chicago); Yu Jerry.3 (Nokia-LC/Alpharetta); Carroll James (Nokia-LC/Malvern); Lieber Dick (Nokia-LC/Chicago); Johnson Anthony (Nokia-LC/Alpharetta); Mccuiston Mitchell (Nokia-LC/Chicago)
Subject: Re: [rabbitmq-discuss] RabbitMQ blocking issue

Peter,

On 12/02/13 19:58, Matthias Radestock wrote:
> [rabbitmqctl report] So there are 7 connections here
>
>> RabbitMQ Netstat [...]
> But why does this only show three connections?
> [...]
> So I think something went wrong with the server-side netstat report.

I figured out what happened...

Your broker is clustered. The 'rabbitmqctl report' shows connections
across the cluster - if you look carefully you'll see that the 'host'
column has entries for two different IP addresses.

But the 'netstat' command was only run on one of the nodes.
Unfortunately the blocked connection we are interested in went to the
other node.

So next time, please run the netstat report on both server nodes.

Regards,

Matthias.

The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit_report
Type: application/octet-stream
Size: 14875 bytes
Desc: rabbit_report
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130213/399a9288/attachment.obj>