[rabbitmq-discuss] Pika 0.9.5 UserWarning: Write buffer exceeded warning threshold

[Gavin M. Roy]

On Thursday, March 31, 2011 at 11:00 PM, can xiang wrote:
> I'm evaluating a messaging system for my web project. I give rabbitmq 2.4.0 and pika 0.9.5 a try, today. It works unbelievably fine. 
> 


Glad to hear it :)

> Below is what I'm doing:
> 
> 1. Use pika with tornado 1.0,  which uses TornadoConnection adapter
>  2. only produce message in tornado handler, no consumer in tornado
> 3. use 3 standalone round-robin worker as consumer
> 
> I did a 1million "hello word" test with "ab -n 1000000 -c 10 http://localhost/helloword". Message flows at 2000+message/s and 100% message receive, all good except I get the following warning in tornado process:
> 
> /usr/local/lib/python2.6/dist-packages/pika-0.9.5-py2.6.egg/pika/connection.py:642: UserWarning: Pika: Write buffer exceeded warning threshold at 1271 bytes and an estimated 21 frames behind
Pika attempts to detect TCP Back-pressure by watching your average frame size over a rolling window and then detecting when the output buffer exceeds the average frame size multiplied by a configurable multiplier which is 10 by default.

http://pika.github.com/connecting.html#tcp-backpressure

What is happening is you're publishing faster than you're consuming (since you're not consuming) and RabbitMQ is throttling you. Pika is raising warnings so you can respond accordingly in your code. If you want to ignore these warnings you can do the following:

from warnings import simplefilter
simplefilter("ignore", "user")

Alternatively you can increase the multiplier to delay when you'll be notified via warning:

connection.set_backpressure_multiplier(100) 

Where 100 is the value you want the multiplier to be.

However. if you do this and put too many messages into RabbitMQ without consuming, you will run into problems with RabbitMQ and your messages will not be sent.

What I would do is warnings.catch_warnings to respond in your application appropriately:

http://docs.python.org/library/warnings.html#warnings.catch_warnings

I hope this helps!

Gavin 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110331/e4a11973/attachment-0001.htm>