[rabbitmq-discuss] Pika reconnection error

[Jason J. W. Williams]

Hi Guys,

Having some issues with a test cluster setup. It's a 3-node cluster
with 2-disk and 1-RAM nodes. The RAM node is joined to both disk
nodes.

rabbit at Phantome - disk
rabbit_1 at Phantome - disk
rabbit_2 at Phantome - RAM

In front of the cluster is an HAProxy instance listening on port 5670
doing simple round-robin TCP load balancing between the cluster
members.

As long as all the nodes are running I can have my consumer connect
through the load balancer to any node and successfully consume.
Likewise with the producer. However, when I shutdown the node the
consumer is currently connected to and allow Pika to do the
reconnection, I get this error:

error: uncaptured python exception, closing channel
<pika.asyncore_adapter.RabbitDispatcher connected at 0x632350> (<class
'pika.exceptions.ChannelClosed'>:Connection.Close(class_id = 0,
method_id = 0, reply_code = 541, reply_text = 'INTERNAL_ERROR')
[/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/asyncore.py|read|74]
[/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/asyncore.py|handle_read_event|413]
[build/bdist.macosx-10.6-universal/egg/pika/asyncore_adapter.py|handle_read|86]
[build/bdist.macosx-10.6-universal/egg/pika/connection.py|on_data_available|268]
[build/bdist.macosx-10.6-universal/egg/pika/connection.py|_login2|375]
[build/bdist.macosx-10.6-universal/egg/pika/connection.py|handle_connection_open|201]
[cluster_test_consumer.py|on_connection_open|53]
[build/bdist.macosx-10.6-universal/egg/pika/spec.py|queue_declare|3003]
[build/bdist.macosx-10.6-universal/egg/pika/channel.py|_rpc|187]
[build/bdist.macosx-10.6-universal/egg/pika/connection.py|_rpc|326]
[build/bdist.macosx-10.6-universal/egg/pika/channel.py|wait_for_reply|125]
[build/bdist.macosx-10.6-universal/egg/pika/channel.py|_ensure|84])

Every few seconds Pika will attempt to reconnect the consumer and the
error repeats. The only unusual error in the Rabbit logs occurs on the
first reconnection: https://gist.github.com/836441

The rest of the time during these errors the logs look like this:
https://gist.github.com/836442

The consumer is disconnecting and reconnecting to HAProxy because the
stats show the connection moving between backend nodes.

The producer continues to operate correctly through the load balancer.

Consumer code is here: https://gist.github.com/836445

Any help/ideas are greatly appreciated.  Thank you in advance.

-J