[rabbitmq-discuss] python client support for channel.flow

Mon Dec 28 21:18:07 GMT 2009

Brian

Have you looked at pika?  http://github.com/tonyg/pika

Sorry to be brief; I hope to reply at greater length in a couple of days.

alexis

On Mon, Dec 28, 2009 at 2:58 AM, Brian Sullivan <bsullivan at lindenlab.com> wrote:
> Hi,
>
> I wanted to follow up on an issue that's been on our radar for a while, but
> I haven't seen any updates on it and wanted to see what our options might be
> here.
>
> We have been leveraging the py-amqplib client library with rabbitmq, which
> has performed quite well in our application with one main limitation.  The
> main issue we have with it is that it does not currently support the
> Channel.Flow message from an AMQP broker - which means when the server tries
> to prevent a producing client app from overloading the system, the client
> ignores the message and continues sending.  Since RabbitMQ will continue to
> process until it allocates all of its memory, it dies a horrible death and
> all of the messages in the broker are lost (reproduction steps below).
>
> Two basic questions:
> 1) What are my alternatives here from a python client side?  I have
> periodically checked into this with Barry for the py-amqplib, but it appears
> to involve a pretty significant code restructuring (if I remember correctly,
> it had to do with the synchronous processing of messages).  We need a fully
> baked client, and Barry's client works well in normal situations - we just
> need it to be more fault-tolerant in the heavy saturation cases to prevent
> production failures.
>
> 2) If there are no viable options from a python client right now, what
> prevents you guys from adding some code to the rabbitmq broker from simply
> failing a connection that refuses to honor the Channel.Flow message when it
> has been broadcast?  I asked this back in 1.5.0 (or around there), and think
> this would be a great safety mechanism to prevent a single rogue client from
> bringing down a broker.
>
> Any help here would be greatly appreciated.  This has bitten us a few times
> in the past and is the weakest part of our pipeline right now.  We are able
> to buffer our messages on disk on the producer side, so any stoppage at the
> rabbitmq side would simply cause our producer app to retry a connection and
> resume transmission.
>
> I don't read this list as actively as I would like, but I didn't find any
> recent posts on this from any searches I was able to come up with.
> Hopefully the answer is out there and I have just missed it.  Please let me
> know if I need to give more info here to help drive the discussion.
>
> Thanks,
> Brian
>
> ==== Reproduction steps ====
>
> Attached is my failure case test script.  In a nutshell, the test is as
> follows:
> - Create non-auto-delete queue, bound to a topic
> - Spam that topic using the attached script until it maxes out the memory on
> the machine (luckily, my test box is only 2GB so this does not take too
> long)
> - You can see the alerts fire in the /var/log/rabbitmq/rabbit.log file, but
> no active flag is set on the Channel object on the Python client side.
> - One note is that I am talking to a 1.5.3 broker, but the Channel.Flow
> should be working fine in that version, this is the client's handling that I
> am focusing on.
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>