<div dir="ltr">We have taken your alternative approach and write large payloads to S3. They go to encrypted private buckets and we generate signed URLs to allow access by downstream components. We employ content distribution networks for some payloads.<div>
<br></div><div>As they are general purpose, we have retooled our messaging pipelines to pass opaque messages, generally compressed binary. It is expected that any content necessary for filtering and routing will be promoted to AMQP headers where it is easily accessible without requiring access to the message body. This has worked quite well in practice with our internal clients.</div>
<div><br></div><div>Like you, we gather a diverse set of inputs. However we distribute payloads globally, sometimes to millions of endpoints, and that affects our approach.</div><div><br></div><div>Michael Laing</div><div>
NYTimes</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 30, 2013 at 4:06 PM, Alexander Schatten <span dir="ltr"><<a href="mailto:alexanderschatten@gmx.at" target="_blank">alexanderschatten@gmx.at</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am considering to use RabbitMQ as messaging platform for a data warehouse (today we would probably call it "Big Data") type application.<br>
<br>
The principle issue is, that we receive data from a large variety of sources; partly batch updates, partly events from sensors and the like. The idea is to have service interfaces for different data sources to the "outside" that accept the data packages and events, enrich it with metadata etc. and then dump it on RabbitMQ queues and topics.<br>
<br>
Consumers are essentially processing units that do statistical analysis, error/warning trigger and storage modules that write raw and aggregated data into databases (PostgreSQL and most likely MongoDB).<br>
<br>
Now, certain messages, e.g. sensor events will be rather small in size. Others like batch updates from ERP and CRM systems or containing documents might be larger; I suppose several MB up to 100 MB. I have not used messaging in such a context, but what I heard is, that other message brokers tend to have problems with large messages or start to behave erratic.<br>
<br>
so my concrete question: has anyone experience with such a use case?<br>
<br>
alternatively it would be possible, to write the (large) payload immediately (at the service interface) into e.g. MongoDB and only put the reference/ID in the message. However, this would break the decoupling to a certain extent as all consumers need access to MongoDB or to a REST interface that serves the payload. Also message filtering and content based routing is limited in the latter case.<br>
<br>
Would appreciate any comments on this issue.<br>
<br>
Kind regards,<br>
<br>
<br>
Alexander Schatten<br>
<br>
<br>
--<br>
==========================================<br>
Dr. Alexander Schatten<br>
==========================================<br>
<a href="http://www.schatten.info" target="_blank">http://www.schatten.info</a><br>
<a href="http://sichten.blogspot.com" target="_blank">http://sichten.blogspot.com</a><br>
Follow me at<br>
<a href="https://alpha.app.net/alex_buzz" target="_blank">https://alpha.app.net/alex_buzz</a><br>
<a href="http://twitter.com/alex_buzz" target="_blank">http://twitter.com/alex_buzz</a><br>
==========================================<br>
"Die Grenzen meiner Sprache<br>
bedeuten die Grenzen meiner Welt.", Wittgenstein<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br></div>