[rabbitmq-discuss] Batching messages

[Srdan Kvrgic]

On Oct 25, 2012, at 12:06 , Emile Joubert wrote:

> Hi,
> 
> On 25/10/12 06:33, Srdan Kvrgic wrote:
>> You think?
> 
> There may be other ways of achieving higher throughput that you should
> explore first before investing in more complex solutions. A potential
> improvement would be to use auto-ack or to acknowledge messages in
> batches. And not to use persistence unless necessary. I assume you have
> exhausted other options.

Auto-acking isn't really an option in production for us, but we do already only ack every 50th message or

> Do you get any throughput improvement if you turn off Nagle's algorithm?
> Both publishing and consuming can be asynchronous when using RabbitMQ.
> If that were not the case then the throughput due to batching would be
> more dramatic. I suspect you will see some improvement due to network
> batching when turning off Nagle's algorithm.

Interesting. I'd never heard of it. Must try it out.

> If the above don't give acceptable gains and you really need batching of
> messages then this can be implemented in a library that sits between
> your application and the broker. Such a library would be in the best
> position to judge the throughput/latency trade-off and limit the batches
> based on size or latency constraints. It should be able to pack messages
> on the publishing side and unpack on the consuming side.
> 
> A possible approach that a library might take to rejecting an individual
> message would be to acknowledge the aggregate batch message and to
> re-publish the rejected message with a suitable header. Message ordering
> will be lost, but this might be a close enough approximation to
> rejecting an individual message in many cases.

That's funny because this is what we've found ourselves building. And it's working. Pretty good. The problem is, every once in a while we'll end up changing the format of a particular message, or have it come string-encoded in a different way. A server might then well parse the message, discover it can't make any sense of it and try to repost it in the screwed up way it parsed it first which leaves the message being constantly requeued.

All this is not really a problem for us. This is not a cry for help or anything dramatic like that. We've got a bunch of libraries in place that give us all the functionality we need. I just thought this feature could be useful for new rabbitmq users allowing them easier batching without laying down weeks of work and a world of gotchas. =)

Kind Regards,
Srdan