[rabbitmq-discuss] Batching messages

[Srdan Kvrgic]

Hi,

Short story:
Are there any plans to support batching of messages within the rabbitmq eco-system?


Long story:
We at Burtcorp are using rabbit-mq to send relatively small messages (1kb) at rates of a couple of thousand per second. We noticed at one point that there seemed to be a lot of overhead with handling individual messages so we started lumping a bunch of messages in a batch and sending that. The throughput sky-rocketed.

I did some benchmarks back in the day (http://goo.gl/wmEmD). The graphs on page 14 and 16 illustrate the extremes of this point. Maxing out sending 10 byte messages at 5000/s or 10kb at 1200/s.

So the throughput boost is great, as could be expected. There's a reason Amazon bundles your books together when shipping them - the overhead of sending each individually would put them out of business.

Still, there are issues with our manual batching. The problem is when we screw up processing a message. The paradigm is to reject the message you can't process but the problem is we got a batch of 10 messages and we only failed processing one. We can't reject the whole batch. That would mean re-processing the 9 messages that got processed the first time around..

Optimally you would want a system that sends and receives batches of messages but allows you to reject single messages. Have your cake and eat it, as it were...

Also automatic de-/compression in the driver. That also was a great performance boost when the batches started getting big. Lzf ftw!

You think?

Kind regards,
Srdan