[rabbitmq-discuss] Batching messages

[Alexis Richardson]

On Fri, Oct 26, 2012 at 11:45 AM, Simon MacMullen <simon at rabbitmq.com> wrote:
> On 26/10/12 11:23, Alexis Richardson wrote:
>>
>> ISTM that basic.publish could be extended to accept a set of messages
>> as the payload parameter in lieu of a single message.  Then all
>> metadata would be the same for all messages in a batch.  In particular
>> all routing would be the same.  This would improve throughput
>> especially when the disk is involved.
>
>
> When the disc is *not* involved.
>
> When the disc *is* involved then its costs are going to dominate everything
> else.

Then please could you explain to me (briefly!)...

a. (all other things being equal eg when everything gets added to the
same queue) whether a 1MB message takes the same time to process as a
batch of 100 10KB messages, 1,000 1KB messages, etc..

b. why any differences may arise




> While such a scheme would benefit in some cases, you'd need quite a specific
> workload to really gain from it (one in which queues do very little work).
> And it would add noticeable complexity. Of course, it would help us lie on
> benchmarks ;-)
>
> So I'm not sure. I may do some experiments on the subject at some point.

I'd be interested in seeing what these reveal.



> Another approach is the entirely client-side approach that Emile mentions,
> and that it sounds like Srdan has written. Srdan, would you like to share?
> :-)

I did not quite understand if that approach was language (and client)
specific or not.