[rabbitmq-discuss] Puka client and publisher confirms performance

[Christos Stavrakakis]

On 03/14/2012 03:16 PM, Marek Majkowski wrote:
> On Wed, Mar 14, 2012 at 13:14, Marek Majkowski<majek04 at gmail.com>  wrote:
>    
>> On Wed, Mar 14, 2012 at 12:31, Christos Stavrakakis
>> <stavr.chris at gmail.com>  wrote:
>>      
>>> I am using Puka client [1] in order to send messages with guaranteed
>>> delivery using publisher confirms.
>>>
>>> Trying to send 200.000 messages I see that waiting for the confirms, takes a
>>> lot of time.
>>>
>>> To be more specific:
>>>
>>>      promises = []
>>>      t0 = time.time()
>>>      for i in range(0,200000):
>>>          promise = client.basic_publish(exchange='test_exchange',
>>> routing_key='test',
>>>                                                                body="Hello
>>> world!")
>>>          promises.append(promise)
>>>
>>>      t1 = time.time()
>>>      client.wait(promises)
>>>      t2 = time.time()
>>>      promise = client.close()
>>>      client.wait(promise)
>>>
>>> In this example t1-t0 is arround 9s while t2-t1 is more than 100 seconds!!
>>> Increasing the number of messages results in even worse performance.
>>>
>>> Can anyone explain this behavior ? Is there a more effective way to use
>>> publisher confirms ?
>>>        
> Right, to the point:
>   - first part of code adds messages to the write buffer - no network activity
>   - second part of the code - wait for replies from rabbit
>     (there may be some cpu cost for managing a big number of promises,
>      within Puka library, this is TBD)
>
> Marek
>    

What is the limit for this buffer ? If understand correct he messages 
are actually sent only when you wait for the promises
or you close the connection. This means that if my connection to a 
server closes before start waiting for promises, I must resent all 
messages ?

Maybe periodically flushing the buffer could speed up things ?

Chris