[rabbitmq-discuss] Large Message Support / Stability
alexanderschatten at gmx.at
Wed Jul 31 08:48:20 BST 2013
thanks for your insight. That is indeed an interesting approach. Our app consumers are most likely not as widely distributed as yours, but the encrypted bucket idea is an interesting one indeed.
However, keeping payload external has further drawbacks, e.g. when can I delete the payload, if at all (that is, how would I know when the last guest has arrived to eat lunch before I remove the buffet *g*)
From a "simplicity" point of view, I would prefer to dump the stuff into the message; however, my gut feeling calls "trouble"!!
thanks a lot for sharing your experiences.
On 30.07.2013, at 23:17, "Laing, Michael" <michael.laing at nytimes.com> wrote:
> We have taken your alternative approach and write large payloads to S3. They go to encrypted private buckets and we generate signed URLs to allow access by downstream components. We employ content distribution networks for some payloads.
> As they are general purpose, we have retooled our messaging pipelines to pass opaque messages, generally compressed binary. It is expected that any content necessary for filtering and routing will be promoted to AMQP headers where it is easily accessible without requiring access to the message body. This has worked quite well in practice with our internal clients.
> Like you, we gather a diverse set of inputs. However we distribute payloads globally, sometimes to millions of endpoints, and that affects our approach.
> Michael Laing
Dr. Alexander Schatten
Follow me at
"Die Selektion hat uns Anschauungsformen gemäß den Aufgaben in noch höchst einfachen Lebensbereichen eingebaut. Und mit Anschauungen von gestern unterwerfen wir uns eine Welt von morgen.", Rupert Riedl
More information about the rabbitmq-discuss