Voldemort has key expiration as well.<br><br><div class="gmail_quote">On Tue, Nov 10, 2009 at 8:29 AM, Kevin A. Smith <span dir="ltr"><<a href="mailto:kevin@hypotheticalabs.com">kevin@hypotheticalabs.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Redis already supports explicit timeouts for entries via the expire and ttl commands so it can take care of that as well.<br>
<font color="#888888"><br>
--Kevin<br>
</font><div><div></div><div class="h5">On Nov 10, 2009, at 10:19 AM, Tony Garnock-Jones wrote:<br>
<br>
> We talk every now and then about deduplication filters, for avoiding<br>
> repeated work. One problem is that it can be hard to know what to do if<br>
> you have a worker pool, because you have to share the state about what<br>
> messages the whole pool has seen, across the whole pool.<br>
><br>
> Sounds like a job for a distributed key-value store. What if redis,<br>
> riak, even memcached, were used as the dedup filter? Each worker pool<br>
> (a.k.a "queue", heh) would have a key-value store of their own. The<br>
> store would hold, for each labelled request, either<br>
><br>
> - nothing: the request is fresh to the pool<br>
><br>
> - an indication of partial completeness and/or receipt<br>
><br>
> - the response that was sent back to the requestor, that can be<br>
> sent back again if a replay is detected, without doing any more<br>
> work<br>
><br>
> The nice thing is that not only are these distributed key-value stores<br>
> pretty quick these days, they also scale up tremendously well and<br>
> furthermore you can "shard" them because you can scope them to the<br>
> collection of workers that are sharing a queue.<br>
><br>
> The only missing piece is clearing out old, definitely stale records<br>
> from the filter after a while. For that you can use a separate<br>
> garbage-collection/expiry process, I guess; I haven't run any<br>
> experiments here, though. Maybe some of these new stores even include<br>
> time-to-live for their records, which solves the problem for us!<br>
><br>
> So: rabbitmq for the routing, buffering, relaying and delivery of<br>
> requests and responses; and a distributed key-value store for deduplication.<br>
><br>
> Tony<br>
><br>
> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
> <a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br>