[rabbitmq-discuss] python client support for channel.flow

[Jason J. W. Williams]

When will Pika implement it's own exceptions? Atm, I'd like to start
using it instead of py-amqplib, but I think I'd rather have a root
canal than reverse engineer all the possible exceptions again (like
you have to with py-ampqlib).

-J

On Mon, Dec 28, 2009 at 3:18 PM, Alexis Richardson
<alexis.richardson at gmail.com> wrote:
> Brian
>
> Have you looked at pika?  http://github.com/tonyg/pika
>
> Sorry to be brief; I hope to reply at greater length in a couple of days.
>
> alexis
>
>
> On Mon, Dec 28, 2009 at 2:58 AM, Brian Sullivan <bsullivan at lindenlab.com> wrote:
>> Hi,
>>
>> I wanted to follow up on an issue that's been on our radar for a while, but
>> I haven't seen any updates on it and wanted to see what our options might be
>> here.
>>
>> We have been leveraging the py-amqplib client library with rabbitmq, which
>> has performed quite well in our application with one main limitation.  The
>> main issue we have with it is that it does not currently support the
>> Channel.Flow message from an AMQP broker - which means when the server tries
>> to prevent a producing client app from overloading the system, the client
>> ignores the message and continues sending.  Since RabbitMQ will continue to
>> process until it allocates all of its memory, it dies a horrible death and
>> all of the messages in the broker are lost (reproduction steps below).
>>
>> Two basic questions:
>> 1) What are my alternatives here from a python client side?  I have
>> periodically checked into this with Barry for the py-amqplib, but it appears
>> to involve a pretty significant code restructuring (if I remember correctly,
>> it had to do with the synchronous processing of messages).  We need a fully
>> baked client, and Barry's client works well in normal situations - we just
>> need it to be more fault-tolerant in the heavy saturation cases to prevent
>> production failures.
>>
>> 2) If there are no viable options from a python client right now, what
>> prevents you guys from adding some code to the rabbitmq broker from simply
>> failing a connection that refuses to honor the Channel.Flow message when it
>> has been broadcast?  I asked this back in 1.5.0 (or around there), and think
>> this would be a great safety mechanism to prevent a single rogue client from
>> bringing down a broker.
>>
>> Any help here would be greatly appreciated.  This has bitten us a few times
>> in the past and is the weakest part of our pipeline right now.  We are able
>> to buffer our messages on disk on the producer side, so any stoppage at the
>> rabbitmq side would simply cause our producer app to retry a connection and
>> resume transmission.
>>
>> I don't read this list as actively as I would like, but I didn't find any
>> recent posts on this from any searches I was able to come up with.
>> Hopefully the answer is out there and I have just missed it.  Please let me
>> know if I need to give more info here to help drive the discussion.
>>
>> Thanks,
>> Brian
>>
>> ==== Reproduction steps ====
>>
>> Attached is my failure case test script.  In a nutshell, the test is as
>> follows:
>> - Create non-auto-delete queue, bound to a topic
>> - Spam that topic using the attached script until it maxes out the memory on
>> the machine (luckily, my test box is only 2GB so this does not take too
>> long)
>> - You can see the alerts fire in the /var/log/rabbitmq/rabbit.log file, but
>> no active flag is set on the Channel object on the Python client side.
>> - One note is that I am talking to a 1.5.3 broker, but the Channel.Flow
>> should be working fine in that version, this is the client's handling that I
>> am focusing on.
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>