[rabbitmq-discuss] Puka client and publisher confirms performance

[Marek Majkowski]

On Wed, Mar 14, 2012 at 12:31, Christos Stavrakakis
<stavr.chris at gmail.com> wrote:
> I am using Puka client [1] in order to send messages with guaranteed
> delivery using publisher confirms.
>
> Trying to send 200.000 messages I see that waiting for the confirms, takes a
> lot of time.
>
> To be more specific:
>
>     promises = []
>     t0 = time.time()
>     for i in range(0,200000):
>         promise = client.basic_publish(exchange='test_exchange',
> routing_key='test',
>                                                               body="Hello
> world!")
>         promises.append(promise)
>
>     t1 = time.time()
>     client.wait(promises)
>     t2 = time.time()
>     promise = client.close()
>     client.wait(promise)
>
> In this example t1-t0 is arround 9s while t2-t1 is more than 100 seconds!!
> Increasing the number of messages results in even worse performance.
>
> Can anyone explain this behavior ? Is there a more effective way to use
> publisher confirms ?

Chris,

It's very cool that you found Puka useful!

Puka does some magic under the hood to track publisher confirms.

First, do make sure you're using publiser confirms, please
add pubacks=True parameter to the Client:

   https://github.com/majek/puka/blob/master/puka/connection.py#L24

(or make sure otherwise that you are using pubacks, there
 should be a property on the client object somewhere....)

Second, here's what I'm doing for performance in the stress_amqp.py example:
https://github.com/majek/puka/blob/master/examples/stress_amqp.py#L54-57

Basically, I'm just making sure the last message went through. This
should give you a reasonable confidence that previous messages
also were delivered.

Finally, it would be nice to identify what is the performance bottleneck
for the code snippet you supplied. It is possible that there's a stupid
performance issue within Puka.

Cheers,
  Marek