[rabbitmq-discuss] Batching messages

[Emile Joubert]

Hi,

On 25/10/12 06:33, Srdan Kvrgic wrote:
> You think?

There may be other ways of achieving higher throughput that you should
explore first before investing in more complex solutions. A potential
improvement would be to use auto-ack or to acknowledge messages in
batches. And not to use persistence unless necessary. I assume you have
exhausted other options.

Do you get any throughput improvement if you turn off Nagle's algorithm?
Both publishing and consuming can be asynchronous when using RabbitMQ.
If that were not the case then the throughput due to batching would be
more dramatic. I suspect you will see some improvement due to network
batching when turning off Nagle's algorithm.

If the above don't give acceptable gains and you really need batching of
messages then this can be implemented in a library that sits between
your application and the broker. Such a library would be in the best
position to judge the throughput/latency trade-off and limit the batches
based on size or latency constraints. It should be able to pack messages
on the publishing side and unpack on the consuming side.

A possible approach that a library might take to rejecting an individual
message would be to acknowledge the aggregate batch message and to
re-publish the rejected message with a suitable header. Message ordering
will be lost, but this might be a close enough approximation to
rejecting an individual message in many cases.



-Emile