[rabbitmq-discuss] RMQ Java Client - ChannelN.waitForConfirm randomly throwing exceptions

• Previous message: [rabbitmq-discuss] RMQ Java Client - ChannelN.waitForConfirm randomly throwing exceptions
• Next message: [rabbitmq-discuss] Exclusive consuming with mirrored queues
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Emanuele,

Bravo for trying AtomicInteger; however the comment you found is an old
one, and probably (I say probably) does not apply anymore. It is
extremely unlikely that a recently used channel number is reallocated
(unless you explicitly request one) and even in that case I think we
have the locking correct now so that this race cannot happen. In any
event, using AtomicInteger will only make things worse.

Incidentally, the code following the comment should detect and fix the
clash, so you shouldn't see it anyhow. (Famous last words.) Of course,
like all naive concurrent code, it does rather assume there are only
ever two threads involved, so could have a very rare bug -- but I doubt
it.

I'm grateful for the extra information; however, it doesn't appear to be
enough to follow what's happening.

It looks as though your application
myapplication.rabbitmq.util.RabbitLinker.send(RabbitLinker.java:262) is
calling waitForConfirms() on the channel. I don't see this in the code
outline you sent. Can you validate that confirmSelect is called on the
channel; that the correct channel is being used top issue
waitForConfirms() and that you issue this call before closing the
channel.

What do you do in the shutdownListener you register, for example?

If it is possible, a look at your application code would be helpful.

Steve Powell  (a happy bunny)
----------some more definitions from the SPD----------
chinchilla (n.) Cooling device for the lower jaw.
socialcast (n.) Someone to whom everyone is speaking but nobody likes.
literacy (n.) A textually transmitted disease usually contracted in childhood.

On 29 Mar 2012, at 09:43, Emanuele G wrote:

> Hello Steve,
> thanks a lot for answering, I really hope we can solve the issue. First of all let's state that I am using RabbitMQ 2.8.1 (and the 2.8.1 java client)
> 
> Now, I tried the AtomicInteger thing because, drilling down into the cause of the exception, I've found this comment in your source code:
> Class: com.rabbitmq.client.impl.ChannelManager
> Method: releaseChannelNumber(ChannelN channel)
>         // Warning, here be dragons. Not great big ones, but little baby ones
>         // which will nibble on your toes and occasionally trip you up when
>         // you least expect it. (Pixies? HP2)
>         // Basically, there's a race that can end us up here. It almost never
>         // happens, but it's easier to repair it when it does than prevent it
>         // from happening in the first place.
>         // If we end up doing a Channel.close in one thread and a Channel.open
>         // with the same channel number in another, the two can overlap in such
>         // a way as to cause disconnectChannel on the old channel to try to
>         // remove the new one. Ideally we would fix this race at the source,
>         // but it's much easier to just catch it here.
> 
> This is exactly what I believe is happening.
> 
> Anyway here are the additional information:
> this is the code that uses the channel (called mainChannel)
>             mainChannel.addShutdownListener(this);
>             mainChannel.basicQos(1);
>             mainChannel.queueDeclare(endpoint.getQueueName(), endpoint.isDurable(),
>                     !EXCLUSIVE, !AUTO_DELETE, args);
> 
>             // Ensure Exchange
>             mainChannel.exchangeDeclare(endpoint.getExchangeName(),
>                     endpoint.getExchangeTypeName(), endpoint.isDurable(),
>                     !EXCLUSIVE, !AUTO_DELETE, args);
> 
>             // Bind Queue-Exchange
>             mainChannel.queueBind(endpoint.getQueueName(),
>                     endpoint.getExchangeName(), endpoint.getRoutingKey());
> 
> 
> This is the complete stacktrace:
> 2012-03-28 15:05:46,057 DEBUG [main] c.b.g.r.u.RabbitLinker [RabbitLinker.java:217] Error while building channel, retrying.
> com.rabbitmq.client.ShutdownSignalException : connection error; reason: {#method<connection.close>(reply-code=503, reply-text=COMMAND_INVALID - second 'channel.open' seen, class-id=20, method-id=10), null, ""}
>        at com.rabbitmq.client.impl.ChannelN.waitForConfirms( ChannelN.java:182)
>        at com.rabbitmq.client.impl.ChannelN.waitForConfirms( ChannelN.java:170)
>        at myapplication.rabbitmq.util.RabbitLinker.send(RabbitLinker.java:262)
>        at myapplication.rabbitmq.producer.AbstractRabbitProducer.send(AbstractRabbitProducer.java:69)
>        at myapplication.rabbitmq.producer.AbstractCamelProducer.process(AbstractCamelProducer.java:64)
>        at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
>        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
>        at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:115)
>        at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:285)
>        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)
>        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
>        at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
>        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
>        at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
>        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
>        at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:322)
>        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:213)
>        at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
>        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
>        at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
>        at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
>        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
>        at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
>        at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
>        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
>        at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
>        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
>        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
>        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
>        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)2012-03-28 15:05:46,060 DEBUG [AMQP Connection 127.0.0.1:5672] c.b.g.r.u.RabbitFactory$ConnectionSaver [RabbitFactory.java:119] shutdownComplete:
> com.rabbitmq.client.ShutdownSignalException : connection error; reason: {#method<connection.close>(reply-code=503, reply-text=COMMAND_INVALID - second 'channel.open' seen, class-id=20, method-id=10), null, ""}
>        at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:654) ~[amqp-client-2.8.1.jar:na]
>        at com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnection.java:610) ~[amqp-client-2.8.1.jar:na]
>        at com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnection.java:584) ~[amqp-client-2.8.1.jar:na]
>        at com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.java:89) ~[amqp-client-2.8.1.jar:na]
>        at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144) ~[amqp-client-2.8.1.jar:na]
>        at com.rabbitmq.client.impl.AMQChannel.handleFrame( AMQChannel.java:91) ~[amqp-client-2.8.1.jar:na]
>        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:509) ~[amqp-client-2.8.1.jar:na]
> 2012-03-28 15:05:46,061 DEBUG [pool-18-thread-2] c.b.g.r.u.RabbitLinker [RabbitLinker.java:322] Internal RabbitMQ Exception while ACK-ing a message,
> 
> 
> I hope you can help me figuring out what is happening.
> 
> Thanks
> 
> 
> > Subject: Re: [rabbitmq-discuss] RMQ Java Client - ChannelN.waitForConfirm randomly throwing exceptions
> > From: steve at rabbitmq.com
> > Date: Wed, 28 Mar 2012 18:17:57 +0100
> > CC: rabbitmq-discuss at lists.rabbitmq.com
> > To: ghepardo_1982 at hotmail.com
> > 
> > Emanuele,
> > 
> > There should be no need to use AtomicInteger to guarantee a new channel
> > number; createChannel should guarantee this for you.
> > 
> > You have shown me the stack trace -- and I can see that your application
> > threads are issuing waitForConfirms().
> > 
> > Can you tell me a little more about how your threads use the Channel?
> > I'd like to know the state of the channel the thread is using, at the
> > time of the call. What happened on that channel immediately before the
> > waitForConfirms() call?
> > 
> > Perhaps some more snippets of the thread code would be useful, and a
> > little more of the stack trace would help.
> > 
> > Finally, was there anything in the rabbit log around this time?
> > 
> > Steve Powell (a happy bunny)
> > ----------some more definitions from the SPD----------
> > chinchilla (n.) Cooling device for the lower jaw.
> > socialcast (n.) Someone to whom everyone is speaking but nobody likes.
> > literacy (n.) A textually transmitted disease usually contracted in childhood.
> > 
> > On 28 Mar 2012, at 17:29, Emanuele Gheradini wrote:
> > 
> > > 
> > > I want also to add that what I understood is:
> > > 
> > > for some reasons RabbitMQ java client gives the same channel (or channel
> > > number) to different threads, then these threads will perform different
> > > operations on that channel which will be closed by RabbitMQjavaclient, thus
> > > resulting in the crash of the waitForConfirm method, at line 182 (here is
> > > the snipped):
> > > 
> > > 181: if (getCloseReason() != null) {
> > > 182: throw Utility.fixStackTrace(getCloseReason());
> > > 183: }
> > > 
> > > 
> > > I tried getting channels from rabbit in this way:
> > > 
> > > AtomicInteger chanNumber = new AtomicInteger(0);
> > > 
> > > connection.createChannel(chanNumber.getAndIncrement());
> > > 
> > > to be sure that each thread gets a channel with a different channelNumber,
> > > but this does not work either...
> > > 
> > > 
> > > I hope someone can give me a suggestion.
> > > 
> > > Thanks.
> > > -- 
> > > View this message in context: http://old.nabble.com/RMQ-Java-Client---ChannelN.waitForConfirm-randomly-throwing-exceptions-tp33544834p33544845.html
> > > Sent from the RabbitMQ mailing list archive at Nabble.com.
> > > 
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-discuss at lists.rabbitmq.com
> > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120329/b3b24d2f/attachment.htm>

• Previous message: [rabbitmq-discuss] RMQ Java Client - ChannelN.waitForConfirm randomly throwing exceptions
• Next message: [rabbitmq-discuss] Exclusive consuming with mirrored queues
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]