[rabbitmq-discuss] RMQ Java Client - ChannelN.waitForConfirm randomly throwing exceptions

Emanuele G ghepardo_1982 at hotmail.com
Thu Mar 29 09:43:58 BST 2012 

Hello Steve,
thanks a lot for answering, I really hope we can solve the issue. First of all let's state that I am using RabbitMQ 2.8.1 (and the 2.8.1 java client)

Now, I tried the AtomicInteger thing because, drilling down into the cause of the exception, I've found this comment in your source code:
Class: com.rabbitmq.client.impl.ChannelManager
Method: releaseChannelNumber(ChannelN channel)
        // Warning, here be dragons. Not great big ones, but little baby ones
        // which will nibble on your toes and occasionally trip you up when
        // you least expect it. (Pixies? HP2)
        // Basically, there's a race that can end us up here. It almost never
        // happens, but it's easier to repair it when it does than prevent it
        // from happening in the first place.
        // If we end up doing a Channel.close in one thread and a Channel.open
        // with the same channel number in another, the two can overlap in such
        // a way as to cause disconnectChannel on the old channel to try to
        // remove the new one. Ideally we would fix this race at the source,
        // but it's much easier to just catch it here.

This is exactly what I believe is happening.

Anyway here are the additional information:
this is the code that uses the channel (called mainChannel)
            mainChannel.addShutdownListener(this);
            mainChannel.basicQos(1);
            mainChannel.queueDeclare(endpoint.getQueueName(), endpoint.isDurable(),
                    !EXCLUSIVE, !AUTO_DELETE, args);

            // Ensure Exchange
            mainChannel.exchangeDeclare(endpoint.getExchangeName(),
                    endpoint.getExchangeTypeName(), endpoint.isDurable(),
                    !EXCLUSIVE, !AUTO_DELETE, args);

            // Bind Queue-Exchange
            mainChannel.queueBind(endpoint.getQueueName(),
                    endpoint.getExchangeName(), endpoint.getRoutingKey());


This is the complete stacktrace:
2012-03-28 15:05:46,057 DEBUG [main] c.b.g.r.u.RabbitLinker [RabbitLinker.java:217] Error while building channel, retrying.
com.rabbitmq.client.ShutdownSignalException : connection error; reason: {#method<connection.close>(reply-code=503, reply-text=COMMAND_INVALID - second 'channel.open' seen, class-id=20, method-id=10), null, ""}
       at com.rabbitmq.client.impl.ChannelN.waitForConfirms( ChannelN.java:182)
       at com.rabbitmq.client.impl.ChannelN.waitForConfirms( ChannelN.java:170)
       at myapplication.rabbitmq.util.RabbitLinker.send(RabbitLinker.java:262)
       at myapplication.rabbitmq.producer.AbstractRabbitProducer.send(AbstractRabbitProducer.java:69)
       at myapplication.rabbitmq.producer.AbstractCamelProducer.process(AbstractCamelProducer.java:64)
       at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
       at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
       at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:115)
       at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:285)
       at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)
       at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
       at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
       at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
       at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
       at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
       at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:322)
       at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:213)
       at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
       at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
       at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
       at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
       at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
       at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
       at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
       at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
       at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
       at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
       at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
       at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
       at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)2012-03-28 15:05:46,060 DEBUG [AMQP Connection 127.0.0.1:5672] c.b.g.r.u.RabbitFactory$ConnectionSaver [RabbitFactory.java:119] shutdownComplete:
com.rabbitmq.client.ShutdownSignalException : connection error; reason: {#method<connection.close>(reply-code=503, reply-text=COMMAND_INVALID - second 'channel.open' seen, class-id=20, method-id=10), null, ""}
       at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:654) ~[amqp-client-2.8.1.jar:na]
       at com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnection.java:610) ~[amqp-client-2.8.1.jar:na]
       at com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnection.java:584) ~[amqp-client-2.8.1.jar:na]
       at com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.java:89) ~[amqp-client-2.8.1.jar:na]
       at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144) ~[amqp-client-2.8.1.jar:na]
       at com.rabbitmq.client.impl.AMQChannel.handleFrame( AMQChannel.java:91) ~[amqp-client-2.8.1.jar:na]
       at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:509) ~[amqp-client-2.8.1.jar:na]
2012-03-28 15:05:46,061 DEBUG [pool-18-thread-2] c.b.g.r.u.RabbitLinker [RabbitLinker.java:322] Internal RabbitMQ Exception while ACK-ing a message,


I hope you can help me figuring out what is happening.

Thanks


> Subject: Re: [rabbitmq-discuss] RMQ Java Client - ChannelN.waitForConfirm randomly throwing exceptions
> From: steve at rabbitmq.com
> Date: Wed, 28 Mar 2012 18:17:57 +0100
> CC: rabbitmq-discuss at lists.rabbitmq.com
> To: ghepardo_1982 at hotmail.com
> 
> Emanuele,
> 
> There should be no need to use AtomicInteger to guarantee a new channel
> number; createChannel should guarantee this for you.
> 
> You have shown me the stack trace -- and I can see that your application
> threads are issuing waitForConfirms().
> 
> Can you tell me a little more about how your threads use the Channel?
> I'd like to know the state of the channel the thread is using, at the
> time of the call. What happened on that channel immediately before the
> waitForConfirms() call?
> 
> Perhaps some more snippets of the thread code would be useful, and a
> little more of the stack trace would help.
> 
> Finally, was there anything in the rabbit log around this time?
> 
> Steve Powell  (a happy bunny)
> ----------some more definitions from the SPD----------
> chinchilla (n.) Cooling device for the lower jaw.
> socialcast (n.) Someone to whom everyone is speaking but nobody likes.
> literacy (n.) A textually transmitted disease usually contracted in childhood.
> 
> On 28 Mar 2012, at 17:29, Emanuele Gheradini wrote:
> 
> > 
> > I want also to add that what I understood is:
> > 
> > for some reasons RabbitMQ java client gives the same channel (or channel
> > number) to different threads, then these threads will perform different
> > operations on that channel which will be closed by RabbitMQjavaclient, thus
> > resulting in the crash of the waitForConfirm method, at line 182 (here is
> > the snipped):
> > 
> > 181: if (getCloseReason() != null) {
> > 182:                  throw Utility.fixStackTrace(getCloseReason());
> > 183:               }
> > 
> > 
> > I tried getting channels from rabbit in this way:
> > 
> > AtomicInteger chanNumber = new AtomicInteger(0);
> > 
> > connection.createChannel(chanNumber.getAndIncrement());
> > 
> > to be sure that each thread gets a channel with a different channelNumber,
> > but this does not work either...
> > 
> > 
> > I hope someone can give me a suggestion.
> > 
> > Thanks.
> > -- 
> > View this message in context: http://old.nabble.com/RMQ-Java-Client---ChannelN.waitForConfirm-randomly-throwing-exceptions-tp33544834p33544845.html
> > Sent from the RabbitMQ mailing list archive at Nabble.com.
> > 
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120329/7d713ee6/attachment.htm>

More information about the rabbitmq-discuss mailing list