[rabbitmq-discuss] Performance issue over time in a consumer using basic qos (mirrored queues)

[Tarjei Romtveit]

After reading some posts on the list it seems like many problems with
performance is nothing new. We are currently reorganizing a cluster
consisting of a myriad of jetty instances to a rabbitmq and a message
driven system. The current component we are moving is shortly a staging
system for a relative slow consumer process..

 Our use-case:.

1. We have currently a small stream of relatively large messages, that
varies in size. (1600 bytes -> 5000 bytes )
2. The incoming rate varies form 0 -> 58 per/s in our current test queues.
3. We have currently two brokers in a cluster
4. The queues are set up in mirrored (We can not lose any messages),
durable and non-exclusive.
5. We use Publisher Confirms/Consumer Cancellation in the client
implementations in Java
6. We use prefetch-count as a basic QOS parameter. We have  tested with: 1,
10 , 15, 25
7. We ack per (deliveryTag % prefetch-count == 0)  to minimize acking.
8. The client can nack a message but it is seldom happening (every 1
million message maybe)

Problem:
Over time we see a larger performance decrease in consuming the messages
from the broker. The consumer is deadly fast in the start. 1-2 ms per
message. Then the retrieval gradually decreases to 25-26 ms per message.
The velocity of the speed decrease is varying by the prefetch size. When
increasing the prefetch count to the time before it reaches 25ms is
exponentially increasing. Up to 12 hours. If we re-initiate the connection
it seem to recover and everything seem OK.

Versions:

Java client: 2.5.1
Broker: 2.5.1 (ubuntu build)

Regards


-- 
Tarjei Romtveit
Data Management Director
Integrasco A/S
Mobile: +47 990 20 074
Address: Hasseldalen 3, N-4878 Grimstad, Norway
www.integrasco.com // twitter.com/integrasco // facebook.com/integrasco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111108/1d5655be/attachment.htm>