[rabbitmq-discuss] Batching messages

[Tim Watson]

Hi

On 10/25/2012 06:33 AM, Srdan Kvrgic wrote:
> Hi,
>
> Short story:
> Are there any plans to support batching of messages within the rabbitmq eco-system?
>
>
> Long story:
[snip]
> Optimally you would want a system that sends and receives batches of messages but allows you to reject single messages. Have your cake and eat it, as it were...

Sounds good, but...

AMQP already provides batching delivery capabilities and these have been 
discussed previously, for example 
https://groups.google.com/forum/?fromgroups=#!topic/rabbitmq-discuss/CidFKmJrBFI. 
But what you seem to be running up against is the fact that your many 
small messages limit throughput, so you want to bundle them up. I would 
suggest putting a processor between the publisher and consumer which 
unpacks the batches and forwards individual messages to the target 
exchange. This could be achieved by writing a custom exchange type, 
configured to understand how to 'unpack' the batch and where to route 
the forwarded messages to. The consumers would then receive these 
individual messages (possibly in batches!) and could reject some or all 
of them, but that rejection would not be apparent to the outside world 
in any way! So you'd need to then change your design somewhat, perhaps 
by putting 'failed' messages onto a different 'error queue' *or* by 
rejecting them and setting up a TTL and Dead Letter eXchange on the 
destination queue so that you can identify failures transparently. In 
either case, you'd want to add some kind of x-batch-id header to the 
'broken up' messages so that your reprocessing procedure is able to 
identify the source batch from which they came.

Doing this 'in the broker' doesn't really make sense to me. How would 
transactions and/or confirms continue to work, for example?

> Also automatic de-/compression in the driver. That also was a great performance boost when the batches started getting big. Lzf ftw!

Hmn, that's an interesting one! Which client library are you using?

> You think?
>
> Kind regards,
> Srdan
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss