[rabbitmq-discuss] : Delayed processing of data

[Laing, Michael]

You can try this:

Lets say you can work on 1 minute boundaries - that's what we do.

When you have a message you want to delay to exchange E, calculate: 1) the
datetime DT to the minute at which you want it released; 2) the
corresponding number of milliseconds MS from now.

Declare a queue QDT with a name incorporating DT and a dead letter exchange
of E. If there is already a queue with that name and exact parameters it's
OK - queue declarations are idempotent so you will neither get a redundant
queue nor will you get an error if it already exists.

Publish your message to queue QDT with an expiry of MS.

At DT (or close enough) the message expirations will take effect and the
messages will be released in queued order to exchange E.

You can use a policy to delete queue QDT.

Hope that helps!

ml


On Fri, Mar 21, 2014 at 6:57 AM, Daniel McGreal <daniel at redbite.com> wrote:

> Ah, I see the problem in the Caveats<http://www.rabbitmq.com/ttl.html#per-message-ttl-caveats>
> .
>
> On 21 Mar 2014, at 10:49, Daniel McGreal wrote:
>
> I'm not familiar with all aspect of the process, but couldn't you set a
> TTL on the message, not the queue?
> http://www.rabbitmq.com/ttl.html#per-message-ttl
> With no consumers on this queue it ends up in ultimately it will be dead
> lettered to an exchange/queue/consumer setup where your actual processing
> happens, i.e. there was at least the specified delay in processing
> customisable per message?
>
> On 21 Mar 2014, at 10:22, Ajinkya Apte wrote:
>
>  Thanks for the reply.  So the only problem I see with what I want to do
> and with whats available in RabbitMQ is I will have to time slice and
> create multiple queues.
> For example:
> One queue with a timeout of 5 min, the other 10 min. Then according to the
> delay time just put incoming data into the proper queue.
> The down side I can see is that new data that will be thrown in a queue
> with say a 10 min timeout, will be theoretically processed anywhere between
> 1-10 mins. I cannot kinda guarantee that the data will be handled only
> after 10 mins.
>
>  Does this make sense to you?
>
>  Do you create a new queue per request coming in, depending on a the
> delay?
>
>  AJ
>
>   From: <Laing>, Michael <michael.laing at nytimes.com>
> Reply-To: Discussions about RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com>
> Date: Thursday, March 20, 2014 6:36 PM
> To: Discussions about RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com>
> Subject: Re: [rabbitmq-discuss] : Delayed processing of data
>
>   We currently use TTLs and Dead Letter Exchanges to accomplish this.
>
>  An app we call Route looks at messages and either:
>
>    1. sends them to an exchange called Distribute, or
>    2. declares a queue with a name reflecting the embargo datetime and a
>    DLX of Distribute and sends the message to it with a TTL calculated to
>    release at that datetime.
>
> This works fine. You can dynamically set a policy to delete the queue.
>
>  Probably we will move away from this in the next few months simply
> because we wish to persist as little as possible in rabbitmq. But it is
> quite simple and functional.
>
>  ml
>
>
> On Thu, Mar 20, 2014 at 2:11 PM, Ajinkya Apte <aapte at zynga.com> wrote:
>
>>  Hey guys,
>> I want to delay the processing of some data coming into a queue. The data
>> has delayTime associated with it.
>>
>>  What I want to do?
>>
>>    1. Data is coming in with a delayTime associated with it.
>>    2. Some data can come with the delayTime=5 mintes from now, some data
>>    with delayTime=10 and so on. This delayTime can be anything.
>>    3. I want to delay the processing of this data.
>>
>>
>>  Questions?
>>
>>    1. From the documentation seems like using message TTL won't be the
>>    way to go, right?
>>    2. Will I have to use the 'Dead Letter Exchanges' concept along with
>>    multiple retry queues having different timeouts?
>>    3. Any other suggestions?
>>
>>
>>  Please let me know.
>> Thanks,
>>
>>  AJ
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>    _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140322/dedf69ed/attachment.html>