[rabbitmq-discuss] RabbitMQ and batch processing

Mon May 19 22:32:36 BST 2014

snip:

>
> So do you persist every messages sent via Fabrik? I think we could do this
> for some things, but we would have to have TTLs for others. We could even
> have our persistence store honor TTLs that are assigned during queue
> creation, I think. I wonder if Cassandra has something built in that would
> make this easy for us.
>
>
Yes. We allow internal clients to assign a 'cache_ttl' for each message.
When we persist the message in Cassandra we set that ttl. By default the
cache_ttl is 3 days, max is 30 days.

The above applies to persistence by 'path' - each message may have multiple
paths - think of a path as a folder for the message that is formatted like
a routing key, e.g. 'feeds-item.breaking-news.1224'. Within each
path/folder messages are stored latest-first by a timeuuid (uuid version
1). Searches for the 'latest' n messages in a path subtree are very
efficient.

Cassandra takes care of deleting them for us and compacting storage - we
never delete manually. Similarly, S3 items are automatically deleted.

We also 'journal' all messages with ttl of 30 days. It has worked well for
us and we can easily scale up and down. The fast generalized cache makes it
possible to support quite interesting apps.

ml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140519/cd5b432b/attachment.html>