[rabbitmq-discuss] Batching messages

Fri Oct 26 14:47:08 BST 2012

On Oct 26, 2012, at 12:45 , Simon MacMullen wrote:

> On 26/10/12 11:23, Alexis Richardson wrote:
>> ISTM that basic.publish could be extended to accept a set of messages
>> as the payload parameter in lieu of a single message.  Then all
>> metadata would be the same for all messages in a batch.  In particular
>> all routing would be the same.  This would improve throughput
>> especially when the disk is involved.
> 
> When the disc is *not* involved.
> 
> When the disc *is* involved then its costs are going to dominate everything else.
> 
> While such a scheme would benefit in some cases, you'd need quite a specific workload to really gain from it (one in which queues do very little work). And it would add noticeable complexity. Of course, it would help us lie on benchmarks ;-)
> 
> So I'm not sure. I may do some experiments on the subject at some point.
> 
> Another approach is the entirely client-side approach that Emile mentions, and that it sounds like Srdan has written. Srdan, would you like to share? :-)

https://github.com/burtcorp/autobahn is a comprehensive project to do all things. It's written in jruby though, so I dunno what good it'll do you. =)

And the solution is very simple, instead of sending message directly, buffer them in a list until there are enough of them. 'Enough' is of course configureable. Once there's enough, BSON|JSON|MsgPack encode the list, compress, send, receive, decompress, decode and Bob's your uncle.

It gets a little complicated when using multiple routing keys. You have to keep lists for each key. But it's not a huge deal. And you have to keep a timeout flusher so you don't keep messages forever. Old batch_publisher was 60-ish lines of ruby, here https://gist.github.com/3958901 

> Cheers, Simon
> 
> -- 
> Simon MacMullen
> RabbitMQ, VMware
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss