[rabbitmq-discuss] Queue consumers are blocked on SocketInputStream.socketRead0

[Nicolas Bosansky]

Hello,
We have a rabbitMQ consumer that takes messages from a queue and write them 
in a HBase store.
When HBase connection fails, the consumer get an exception and messages are 
not acknowledged, that's ok.
After a few minutes, the consumer doesn't get messages anymore, even after 
restarting Hbase.

When debugging rabbitMQ client, we have noticed, that it is blocked on the 
following line: SocketInputStream:129 ( Java build 1.6.0_33-b03)
The Frame class is waiting for messages, but the used socket doesn't 
receive any data anymore.
We also notice, that there is no timeout on this socket.

  public static Frame readFrom(DataInputStream is) throws IOException {
        int type;
        int channel;

        try {
            type = is.readUnsignedByte();
        } catch (SocketTimeoutException ste) {
            // System.err.println("Timed out waiting for a frame.");
            return null; // failed
        }
        ....
}

Here is my rabbitMQ report : 

Reporting server status on {{2012,10,18},{15,39,34}}

Status of node rabbit at localhost
[{pid,20127},
 {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
                        {os_mon,"CPO  CXC 138 46","2.2.7"},
                        {sasl,"SASL  CXC 138 11","2.1.10"},
                        {mnesia,"MNESIA  CXC 138 12","4.5"},
                        {stdlib,"ERTS  CXC 138 10","1.17.5"},
                        {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] 
[rq:4] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,47487536},
          {processes,12533768},
          {processes_used,12228104},
          {system,34953768},
          {atom,1183993},
          {atom_used,1157601},
          {binary,16573424},
          {code,11134417},
          {ets,4108160}]},
 {vm_memory_high_watermark,0.39999999990305685},
 {vm_memory_limit,3300904140}]

Cluster status of node rabbit at samcrow ...
[{nodes,[{disc,[rabbit at samcrow]}]},{running_nodes,[rabbit at samcrow]}]

Application environment of node rabbit at samcrow ...
[{auth_backends,[rabbit_auth_backend_internal]},
 {auth_mechanisms,['PLAIN','AMQPLAIN']},
 {backing_queue_module,rabbit_variable_queue},
 {cluster_nodes,[]},
 {collect_statistics,none},
 {collect_statistics_interval,5000},
 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
 {default_user,<<"guest">>},
 {default_user_tags,[administrator]},
 {default_vhost,<<"/">>},
 {delegate_count,16},
 {error_logger,{file,"/var/log/rabbitmq/rabbit at samcrow.log"}},
 {frame_max,131072},
 {hipe_compile,false},
 {included_applications,[]},
 {msg_store_file_size_limit,16777216},
 {msg_store_index_module,rabbit_msg_store_ets_index},
 {queue_index_max_journal_entries,262144},
 {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at samcrow-sasl.log"}},
 {server_properties,[]},
 {ssl_listeners,[]},
 {ssl_options,[]},
 {tcp_listen_options,[binary,
                      {packet,raw},
                      {reuseaddr,true},
                      {backlog,128},
                      {nodelay,true},
                      {exit_on_close,false}]},
 {tcp_listeners,[5672]},
 {trace_vhosts,[]},
 {vm_memory_high_watermark,0.4}]

Connections:
pid address port peer_address peer_port ssl peer_cert_subject 
peer_cert_issuer peer_cert_validity auth_mechanism ssl_protocol 
ssl_key_exchange ssl_cipher ssl_hash protocol user vhost timeout frame_max 
client_properties recv_oct recv_cnt send_oct send_cnt send_pend state 
channels
<rabbit at samcrow.2.2883.0> 127.0.0.1 5672 127.0.0.1 15220 false PLAIN {0,9,1} 
guest / 0 131072[{"product","RabbitMQ"},{"information","Licensed under the 
MPL. See 
http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright 
(C) 2007-2012 VMware, Inc."},{"version","2.8.1"}] 11864749 27712 26676264 
11607 0 running 23

I use rabbitMQ client version 2.8.1.

Do you have any idea ?
Thanks in advance,
Nicolas




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121018/eb96cc48/attachment.htm>