[rabbitmq-discuss] Rabbitmq-Shovel Timer-Based Publishing Possible?

[Matthias Radestock]

Allen,

Allen Kim wrote:
> Message volume fluctuates from extremely high to very low depending on the
> time of day.
> 
> We use a relatively large transaction size on the outbound, publishing,
> connection to get high enough throughput on a high latency link for the
> peak times of day.  However, at the off peak times of day, it can take up
> to a few minutes for the shovel to accumulate enough messages to hit the
> configured transaction commit size.
> 
> Allowing a configurable transaction size plus commit timer adds some
> flexibility.  At peak times, the transaction size counter is always hit
> first.  At low times, the timer hits first and ensures that  message
> delivery does not stall.

So you are employing transactions in order to get guaranteed delivery, 
right?

I can see at least two alternative approaches here...

1) instead of committing after a certain size/time, grow the tx as long 
as there are messages and only commit when there is a gap.

2) use confirms (http://www.rabbitmq.com/extensions.html#confirms) and 
only ack a message to the source once it has been confirmed by the 
destination.

Both approaches require zero config - no obscure time/count knobs to 
tweak - which makes them rather appealing.

Unfortunately the shovel currently doesn't support either of these. I've 
filed a bug to look into the latter.


Regards,

Matthias.