[rabbitmq-discuss] pika behavior when server die

[can xiang]

Hi,

I'm testing rabbitmq pubsub pattern with pika 0.9.5. Basically what I do are
the following:

1. publish message in a tornado application
2. consume message in a standalone script
3. ab test the tornado application to see how messages goes

basic parameters used:

1. exchange_type = fanout
2. auto_delete = False
3. durable = True
4. delivery_mode = 2
5. no_ack = True
6. SimpleReconnectionStrategy for reconnection

When I'm testing the server die scenario. I found three issues:

ISSUE 1:
=======
Stop rabbitmq server, tornado application raises the following exception and
become not responsive:

2011-04-23 11:48:53,872 - ERROR - TornadoConnection: Socket Error on 9: 104
2011-04-23 11:48:53,872 - ERROR - Exception in I/O handler for fd 9
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/dist-packages/tornado-1.1-py2.6.egg/tornado/ioloop.py",
line 272, in start
    self._handlers[fd](fd, events)
  File
"/usr/local/lib/python2.6/dist-packages/tornado-1.1-py2.6.egg/tornado/stack_context.py",
line 128, in wrapped
    callback(*args, **kwargs)
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py",
line 137, in _handle_events
    self._handle_error(error)
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py",
line 106, in _handle_error
    error_code = error[0]  # Python <= 2.5
TypeError: 'NoneType' object is unsubscriptable
2011-04-23 11:48:53,886 - ERROR - Exception in callback <functools.partial
object at 0x9e500f4>
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/dist-packages/tornado-1.1-py2.6.egg/tornado/ioloop.py",
line 333, in _run_callback
    callback()
  File
"/usr/local/lib/python2.6/dist-packages/tornado-1.1-py2.6.egg/tornado/stack_context.py",
line 128, in wrapped
    callback(*args, **kwargs)
  File "/home/ken/workspace/QvodGround2/utils/messaging.py", line 53, in
connect
    reconnection_strategy=srs)
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/adapters/tornado_connection.py",
line 33, in __init__
    reconnection_strategy)
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py",
line 50, in __init__
    reconnection_strategy)
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/connection.py",
line 170, in __init__
    self._connect()
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/connection.py",
line 228, in _connect
    self.parameters.port or  spec.PORT)
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/adapters/tornado_connection.py",
line 39, in _adapter_connect
    BaseConnection._adapter_connect(self, host, port)
  File
"/usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py",
line 58, in _adapter_connect
    self.socket.connect((host, port))
  File "<string>", line 1, in connect
error: [Errno 111] Connection refused

ISSUE2:
=======
I tested 800 messages, with slow consumer. while it runs, stop
rabbitmq-server with Ctrl+C, consumer is able to continuing consuming
message and finish all 800 messages, then it EXIT.

It's very weird indeed, because the server already gone. Is there any
incoming message buffer in pika? How many messages can be buffered? Are
those message stored in memory? Is it possible to configure the memory
usage?

ISSUE3:
=======
In my use case, it's very important the tornado application wouldn't die for
any reason. If there is anything bad happened to the rabbitmq-server or
connection, messaging system can be down-graded as a not functioning part,
which can go back automatically when server issues resolved. The long
running consuming process shouldn't die or exit when server died or
connection broken. Is this requirement sensible enough?

I hope SimpleReconnectionStrategy works for my case, which is not. Is there
anything wrong in my application code? Or I just roll my own one, any
suggestion on how to do it?

I appreciate any help.

Best regards!
can

PS:  partial tornado application code can be found at gist:
https://gist.github.com/938517 , consumer script:
https://gist.github.com/938226
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110423/5387469f/attachment-0001.htm>