[rabbitmq-discuss] RabbitMQ and batch processing

[Greg Poirier]

On Mon, May 19, 2014 at 2:32 PM, Laing, Michael
<michael.laing at nytimes.com>wrote:

> snip:
>
>>
>> So do you persist every messages sent via Fabrik? I think we could do
>> this for some things, but we would have to have TTLs for others. We could
>> even have our persistence store honor TTLs that are assigned during queue
>> creation, I think. I wonder if Cassandra has something built in that would
>> make this easy for us.
>>
>>
> Yes. We allow internal clients to assign a 'cache_ttl' for each message.
> When we persist the message in Cassandra we set that ttl. By default the
> cache_ttl is 3 days, max is 30 days.
>
> The above applies to persistence by 'path' - each message may have
> multiple paths - think of a path as a folder for the message that is
> formatted like a routing key, e.g. 'feeds-item.breaking-news.1224'. Within
> each path/folder messages are stored latest-first by a timeuuid (uuid
> version 1). Searches for the 'latest' n messages in a path subtree are very
> efficient.
>

Aha! This is really cool!

I like the idea of being able to keep a large "buffer" so to speak of
recently sent messages available for analysis. So, a failed batch job could
very easily inspect the actual messages that were persisted/sent in an
effort to debug. That is pretty sweet. Cassandra is very nice for this as
well, since persistence to C* is can be delegated to a consumer in the
buffer layer or core. Do you simply shovel everything to a particular path
queue for something to persist to Cassandra? Or is that done inline
somewhere before publishing to RabbitMQ? I assume that that kind of thing
is done out-of-band of the client.


> We also 'journal' all messages with ttl of 30 days. It has worked well for
> us and we can easily scale up and down. The fast generalized cache makes it
> possible to support quite interesting apps.
>

Hmm. I'm not sure I follow. What do you mean by journal?

I am also curious about some of the things you do with Fabrik... For
example, is persistence handled by the client (indirectly via the Fabrik
library) or some other component service that listens to RabbitMQ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140519/7aa78a32/attachment.html>