[rabbitmq-discuss] Batching messages
tim at rabbitmq.com
Thu Oct 25 10:54:17 BST 2012
On 10/25/2012 06:33 AM, Srdan Kvrgic wrote:
> Short story:
> Are there any plans to support batching of messages within the rabbitmq eco-system?
> Long story:
> Optimally you would want a system that sends and receives batches of messages but allows you to reject single messages. Have your cake and eat it, as it were...
Sounds good, but...
AMQP already provides batching delivery capabilities and these have been
discussed previously, for example
But what you seem to be running up against is the fact that your many
small messages limit throughput, so you want to bundle them up. I would
suggest putting a processor between the publisher and consumer which
unpacks the batches and forwards individual messages to the target
exchange. This could be achieved by writing a custom exchange type,
configured to understand how to 'unpack' the batch and where to route
the forwarded messages to. The consumers would then receive these
individual messages (possibly in batches!) and could reject some or all
of them, but that rejection would not be apparent to the outside world
in any way! So you'd need to then change your design somewhat, perhaps
by putting 'failed' messages onto a different 'error queue' *or* by
rejecting them and setting up a TTL and Dead Letter eXchange on the
destination queue so that you can identify failures transparently. In
either case, you'd want to add some kind of x-batch-id header to the
'broken up' messages so that your reprocessing procedure is able to
identify the source batch from which they came.
Doing this 'in the broker' doesn't really make sense to me. How would
transactions and/or confirms continue to work, for example?
> Also automatic de-/compression in the driver. That also was a great performance boost when the batches started getting big. Lzf ftw!
Hmn, that's an interesting one! Which client library are you using?
> You think?
> Kind regards,
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
More information about the rabbitmq-discuss