[rabbitmq-discuss] Rabbit MQ: Improve queue flushing speed

[Tim Watson]

BTW - if you're using the java client, there is a perf test tool you can run to check out your configuration - called MulticastMain and you can run it via the 'scripts' directory of the binary distribution. Check out runperftest.sh for examples of how its used, and runjava.sh to execute it.

Please give this a try to see what kind of throughput the client can get in your environment.

Cheers,
Tim 

On 28 Mar 2013, at 15:26, Tim Watson wrote:

> Priyank,
> 
> On 28 Mar 2013, at 12:33, Priyank Patil wrote:
> 
>> The client is unable to fetch those messages at that rate. As a result the queue on the server keeps growing.
>> 
> [snip]
>> Each message is less than 1 KB and I have a healthy 2 Mbps line between the server and my machine. Using a network monitoring utility, I found that it is hardly using any of that bandwidth.
>> 
>> The client is doing nothing with the messages as of now, just printing them to console so processing time on client is almost 0.
>> 
> 
> So why is the client 'unable to fetch those messages at that rate' then? Printing data to the console is not 0 processing time, although I agree it shouldn't be pushing your consumers down to 10 mps. Even with explicit ACKs I'd expect a consumer to manage in the order of KHz when consuming.
> 
>> Some other details: I am using a java client. I have set the client to prefetch 10000 messages. (also tried with default values) The round trip time is about 290 - 350 ms. Messages are individually acknowledged.
>> 
> 
> What do you mean 'round trip time' exactly? You are publishing and consuming from different channels right? You might find reducing the prefetch count a bit actually helps - 10k is quite a high setting. You might also like to confirm in batches (using the 'multiple' flag to indicate that all messages up to the current delivery-tag are being acknowledged.
> 
>> The available resources are being underutilized and 10 messages per second is hardly any load in my opinion. How do I speed up things so that messages held up in queue are transferred faster to the client. Possibly using some sort of batching.
>> 
> 
> Batching ACKs might help, as might reducing the prefetch count to something a bit more sensible (like 100s or 1000s for example). I suspect there's something else, more fundamental that's wrong with your setup, as these figures are abysmally slow and Rabbit is perfectly capable of handling much higher throughput for both producers and consumers, even with all the guarantees and safety measures like persistence, confirms and explicit ACKs turned on.
> 
> Cheers,
> Tim