[rabbitmq-discuss] Rabbit MQ: Improve queue flushing speed

[François Beausoleil]

Le 2013-03-28 à 08:33, Priyank Patil a écrit :

> I have a durable queue which holds persistent messages. The messages arrive into the queue at a rate of about 10 messages per second.
> 
> The client is unable to fetch those messages at that rate. As a result the queue on the server keeps growing.
> 
> Each message is less than 1 KB and I have a healthy 2 Mbps line between the server and my machine. Using a network monitoring utility, I found that it is hardly using any of that bandwidth.
> 
> The client is doing nothing with the messages as of now, just printing them to console so processing time on client is almost 0.
> 
> Some other details: I am using a java client. I have set the client to prefetch 10000 messages. (also tried with default values) The round trip time is about 290 - 350 ms. Messages are individually acknowledged.
> 
> The available resources are being underutilized and 10 messages per second is hardly any load in my opinion. How do I speed up things so that messages held up in queue are transferred faster to the client. Possibly using some sort of batching.
> 
I regularly process 2000 messages per second from durable queues with persistent messages, with a prefetch-count of 2000. My messages are 1500 to 2000 bytes each. The RabbitMQ server and the consumer are both in the same data center. My consumer is writing the messages to disk, in a flat-file.

There must be something in the Java code which causes the slowdown. Can you post your setup and processing loop?

Bye,
François

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130328/b8d62b5e/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4142 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130328/b8d62b5e/attachment.bin>