[rabbitmq-discuss] Rabbitmq-Shovel Timer-Based Publishing Possible?

[Allen Kim]

Replying in the context

On 11-03-16 6:00 PM, "Matthias Radestock" <matthias at rabbitmq.com> wrote:


>Allen,
>
>Allen Kim wrote:
>> Message volume fluctuates from extremely high to very low depending on
>>the
>> time of day.
>> 
>> We use a relatively large transaction size on the outbound, publishing,
>> connection to get high enough throughput on a high latency link for the
>> peak times of day.  However, at the off peak times of day, it can take
>>up
>> to a few minutes for the shovel to accumulate enough messages to hit the
>> configured transaction commit size.
>> 
>> Allowing a configurable transaction size plus commit timer adds some
>> flexibility.  At peak times, the transaction size counter is always hit
>> first.  At low times, the timer hits first and ensures that  message
>> delivery does not stall.
>
>So you are employing transactions in order to get guaranteed delivery,
>right?

Sorry, I am not talking about guaranteed delivery, but I am talking about
delivery on time.

Think about this situation,

Data center A has a transaction, which must be passed to data center B IN
5 seconds.

I haven't thought about guaranteed delivery that you mentioned, which
sounds really good.



>
>I can see at least two alternative approaches here...
>
>1) instead of committing after a certain size/time, grow the tx as long
>as there are messages and only commit when there is a gap.

I don't get "when there is a gap". Gap between what?
>
>2) use confirms (http://www.rabbitmq.com/extensions.html#confirms) and
>only ack a message to the source once it has been confirmed by the
>destination.
>
>Both approaches require zero config - no obscure time/count knobs to
>tweak - which makes them rather appealing.
>
>Unfortunately the shovel currently doesn't support either of these. I've
>filed a bug to look into the latter.
>
>
>Regards,
>
>Matthias.