[rabbitmq-discuss] Puka client and publisher confirms performance

Wed Mar 14 13:45:51 GMT 2012

On 03/14/2012 03:14 PM, Marek Majkowski wrote:

>  On Wed, Mar 14, 2012 at 12:31, Christos Stavrakakis
>  <stavr.chris at gmail.com>   wrote:
>
>>  I am using Puka client [1] in order to send messages with guaranteed
>>  delivery using publisher confirms.
>>
>>  Trying to send 200.000 messages I see that waiting for the confirms, takes a
>>  lot of time.
>>
>>  To be more specific:
>>
>>       promises = []
>>       t0 = time.time()
>>       for i in range(0,200000):
>>           promise = client.basic_publish(exchange='test_exchange',
>>  routing_key='test',
>>                                                                 body="Hello
>>  world!")
>>           promises.append(promise)
>>
>>       t1 = time.time()
>>       client.wait(promises)
>>       t2 = time.time()
>>       promise = client.close()
>>       client.wait(promise)
>>
>>  In this example t1-t0 is arround 9s while t2-t1 is more than 100 seconds!!
>>  Increasing the number of messages results in even worse performance.
>>
>>  Can anyone explain this behavior ? Is there a more effective way to use
>>  publisher confirms ?
>>
>  Chris,
>
>  It's very cool that you found Puka useful!
>
>
Puka is indeed a very practical and beautiful library. It also makes working
with publisher confirms very straightforward. I would recommend to
anyone who
wants a python client for RabbitMQ to give it a try!
>  Puka does some magic under the hood to track publisher confirms.
>
>  First, do make sure you're using publiser confirms, please
>  add pubacks=True parameter to the Client:
>
>      https://github.com/majek/puka/blob/master/puka/connection.py#L24
>
>  (or make sure otherwise that you are using pubacks, there
>    should be a property on the client object somewhere....)
>
>
Yes I am passing pubacks=True parameter to the client.
>  Second, here's what I'm doing for performance in the stress_amqp.py example:
>  https://github.com/majek/puka/blob/master/examples/stress_amqp.py#L54-57
>
>  Basically, I'm just making sure the last message went through. This
>  should give you a reasonable confidence that previous messages
>  also were delivered.
>
It is reasonable, but I believe that it is not guaranteed. Am i right? I
want to make sure that my messages are delivered, but without the penalty
of waiting an acknowledge for each message.

Does puka takes advantage of the multiple field in basic.acks ? If not,
maybe this could speed up the process!
>  Finally, it would be nice to identify what is the performance bottleneck
>  for the code snippet you supplied. It is possible that there's a stupid
>  performance issue within Puka.
>
>  Cheers,
>     Marek
>

Chris