[rabbitmq-discuss] Can RabbitMQ handle big messages?

[Irmo Manie]

RabbitMQ should actually not be used for big file transfers or only
with great care and fragmenting the files into smaller separate
messages.
When running a single broker instance, you'd still be safe, but in a
clustered setup, very big messages will break the cluster.

Clustered nodes are connected via 1 tcp connection, which must also
transport a (erlang) heartbeat. If your big message takes more time to
transfer between nodes than the heartbeat timeout (anywhere between
~20-45 seconds if I'm correct), the cluster will break and your
message is lost.

The preferred architecture for file transfer over amqp is to just send
a message with a link to a downloadable resource and let the file
transfer be handle by specialized protocol like ftp :-)


- Irmo

On Mon, Mar 12, 2012 at 1:26 PM, Emile Joubert <emile at rabbitmq.com> wrote:
> Hi Zabrane,
>
> On 10/03/12 18:34, Jerry Kuch wrote:
>> Hi, Zabrane:  In theory the AMQP protocol IIRC allows crazy large
>> message payloads (2^64 bytes I believe).  In practice though, that's
>> madness since you end up with potential copying and buffering along the
>> way that could make a broker very unhealthy.
>
> I have successfully processed messages as large as 2Gb using RabbitMQ,
> where 2Gb was about 5% of the total RAM. If the ratio between message
> size and total RAM stays low then you can send even larger messages, up
> to the limit Jerry mentioned.
>
>
> -Emile
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss