[rabbitmq-discuss] Java client - com.rabbitmq.client.AlreadyClosedException and Shutdown event

Wed Sep 18 12:55:21 BST 2013

Hello,

Since a week ago, I was having a lot of 
"com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; 
reason: Attempt to use closed channel" the worst part was that the 
"DefaultExceptionHandler" was caught without being rethrowed, so I was 
unable to automatically restart my app when this occured. I updated to the 
last amqp-java client version but nothing changed.

So I extended AMQConnection to specify my own exception handler in order to 
crash the app (with System.exit()) when this occured.

public ExtendedAMQConnection(ConnectionFactory factory, Integer 
workerCount) throws Exception {
        super(factory.getUsername(),
                factory.getPassword(),
                new 
SocketFrameHandler(factory.getSocketFactory().createSocket(factory.getHost(), 
factory.getPort())),
                Executors.newFixedThreadPool(workerCount),
                factory.getVirtualHost(),
                factory.getClientProperties(),
                factory.getRequestedFrameMax(),
                factory.getRequestedChannelMax(),
                factory.getRequestedHeartbeat(),
                factory.getSaslConfig(),
                new DefaultExceptionHandler() {
                    @Override
                    public void handleConsumerException(Channel channel,
                                                        Throwable exception,
                                                        Consumer consumer,
                                                        String consumerTag,
                                                        String methodName) {
                        try {
                            // TODO: change this to call 4-parameter close 
and make 6-parm one private
                            ((AMQConnection) channel.getConnection())
                                    .close(AMQP.INTERNAL_ERROR,
                                            "Internal error in Consumer " + 
consumerTag,
                                            false,
                                            exception,
                                            -1,
                                            false);
                        } catch (Throwable e) {
                            console.error("handleConsumerException", e);
                            System.exit(1);
                        }
                    }
                });
        this.start();
    }

But after some time I understood that it was not the main issue, the app 
was still throwing the same error without doing anything at all to fix it 
so I added a listener to channel shutdown event:

Connection conn = new BringrAMQConnection(factory, workerCount);
Channel channel = conn.createChannel();
channel.addShutdownListener(new ShutdownListener() {
            @Override
            public void shutdownCompleted(ShutdownSignalException e) {
                console.error("RabbitMQ[channel] Shutdown. Stopping the 
server");
                System.exit(1);
            }
        });

and now I got a lot of restarts. I'm not sure I clearly understood if I 
should do something when "shutdownCompleted" occurs (I never stopped 
RabbitMQ and RabbitMQ was always running when this happened), or if 
amqp-client can handle this by itself.

I use com.rabbitmq:amqp-client:3.1.4 with a cluster of two mirrored 
RabbitMQ v3.1.1.

# rabbitmqctl status
Status of node rabbit at ir ...
[{pid,19842},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.1.1"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.1"},
      {rabbit,"RabbitMQ","3.1.1"},
      {os_mon,"CPO  CXC 138 46","2.2.9"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.1"},
      {webmachine,"webmachine","1.9.1-rmq3.1.1-git52e62bc"},
      {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.1-gitd541e9a"},
      {xmerl,"XML parser","1.3.1"},
      {inets,"INETS  CXC 138 49","5.9"},
      {amqp_client,"RabbitMQ AMQP Client","3.1.1"},
      {mnesia,"MNESIA  CXC 138 12","4.7"},
      {sasl,"SASL  CXC 138 11","2.2.1"},
      {stdlib,"ERTS  CXC 138 10","1.18.1"},
      {kernel,"ERTS  CXC 138 10","2.15.1"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:24:24] 
[async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,122151960},
      {connection_procs,22551824},
      {queue_procs,9699000},
      {plugins,415976},
      {other_proc,9745810},
      {mnesia,650608},
      {mgmt_db,10643096},
      {msg_index,431544},
      {other_ets,2452056},
      {binary,35320928},
      {code,18521111},
      {atom,2525529},
      {other_system,9194478}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,27064667340},
 {disk_free_limit,1000000000},
 {disk_free,2864417656832},
 {file_descriptors,
     [{total_limit,924},
      {total_used,261},
      {sockets_limit,829},
      {sockets_used,227}]},
 {processes,[{limit,1048576},{used,2804}]},
 {run_queue,0},
 {uptime,8096817}]

I'm able send you privately the output of rabbitmqctl report if you need it.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130918/931452a5/attachment.htm>