[rabbitmq-discuss] Pika write buffer problem

Mon May 28 13:30:11 BST 2012

I had the same issue, depending on the meg size and network layout it can publish like 1’000msg, and then pika crashes with publishing the like 10’000th message. It will continue to publish messages (no matter whether actually sent to the broker or not) until it hits the end of the buffer.

In order to avoid this either do not publish too fast, implement the TCP_Backpressure CB to let the buffer drain, or use transactional channels which seems to avoid this issue at all.

Cheers
josh

From: rabbitmq-discuss-bounces at lists.rabbitmq.com [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Gavin M. Roy
Sent: Sonntag, 27. Mai 2012 23:01
To: James Dear
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Pika write buffer problem

The warnings are just warnings, letting you know that the writes to rabbitMQ are not happening as quickly as you'd like them to do. Are you saying they are not actually getting into RabbitMQ?

On Sunday, May 27, 2012 at 4:27 PM, James Dear wrote:
Hi,

I'm having trouble getting simple examples of basic_publish to work due to the write buffer being exceeded. If I use the demo_send code from here<http://pika.github.com/examples.html>, I only manage to send 5 messages before getting write buffer exceeded problems. Here's my output:

[INFO     2012-05-27 20:30:16,863  #5451] Connecting fd 3 to 127.0.0.1:5672<http://127.0.0.1:5672>
[INFO     2012-05-27 20:30:17,869  #5451] demo_send: Connected to RabbitMQ
[INFO     2012-05-27 20:30:17,872  #5451] demo_send: Received our Channel
[INFO     2012-05-27 20:30:17,882  #5451] demo_send: Queue Declared
[INFO     2012-05-27 20:30:17,883  #5451] Sending: Hello World #0: 1338147017.88308692
[INFO     2012-05-27 20:30:17,883  #5451] Sending: Hello World #1: 1338147017.88367391
[INFO     2012-05-27 20:30:17,884  #5451] Sending: Hello World #2: 1338147017.88425899
[INFO     2012-05-27 20:30:17,884  #5451] Sending: Hello World #3: 1338147017.88472700
[INFO     2012-05-27 20:30:17,885  #5451] Sending: Hello World #4: 1338147017.88527107
/..snip../pika/connection.py:766: UserWarning: Pika: Write buffer exceeded warning threshold at 413 bytes and an estimated 10 frames behind
  warn(message % (self.outbound_buffer.size, est_frames_behind))
/..snip../pika/connection.py:766: UserWarning: Pika: Write buffer exceeded warning threshold at 447 bytes and an estimated 11 frames behind
  warn(message % (self.outbound_buffer.size, est_frames_behind))
/..snip../pika/connection.py:766: UserWarning: Pika: Write buffer exceeded warning threshold at 490 bytes and an estimated 12 frames behind
  warn(message % (self.outbound_buffer.size, est_frames_behind))
[INFO     2012-05-27 20:30:17,887  #5451] Sending: Hello World #5: 1338147017.88733912
/..snip../pika/connection.py:766: UserWarning: Pika: Write buffer exceeded warning threshold at 511 bytes and an estimated 13 frames behind
  warn(message % (self.outbound_buffer.size, est_frames_behind))
/..snip../pika/connection.py:766: UserWarning: Pika: Write buffer exceeded warning threshold at 545 bytes and an estimated 14 frames behind
  warn(message % (self.outbound_buffer.size, est_frames_behind))
/..snip../pika/connection.py:766: UserWarning: Pika: Write buffer exceeded warning threshold at 588 bytes and an estimated 15 frames behind
  warn(message % (self.outbound_buffer.size, est_frames_behind))

etc etc

On the other hand, everything works fine with pika twisted and the msgs do end up on RabbitMQ with pika twisted, so it doesn't seem to be a restriction to do with RabbitMQ per se.

I've created gists<https://gist.github.com/2815757> for the pika native code which doesn't work and the pika twisted code that does work.

If anyone could help, I would greatly appreciate it.

James
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

--

----

ASG at hnet
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120528/ff352cf8/attachment.htm>