[rabbitmq-discuss] Problem transferring large message

[Matthias Radestock]

Zabrane,

On 22/08/12 12:22, Zabrane Mickael wrote:
>> On 22/08/12 10:34, Zabrane Mickael wrote:
>>> I asked this question long time ago and the idea was to break our
>>> big messages (~50Mb) in smaller ones (100Kb). Everything was fine
>>> after that with the default settings !!!
>>
>> Fragmenting the message is fine, but the broker does not force you
>> to do this.
>
> You sure about stability? Sending 50Mb at high rate makes RabbitMQ
> very unhappy (lot of crashes and messages lost).

I have just tested twenty producers sending 50Mb messages to Rabbit as
fast as they can. The combined rate was ~30Hz. And that's on a fairly 
modest machine. There were some long pauses when messages were getting 
paged to disk, and memory peaked at about 80% (so twice the configured 
threshold), but that's expected. No crashes.

With smaller messages the performance profile will be smoother, i.e. the 
pauses will be shorter and memory peaks will be lower. But there are 
downsides to fragmenting messages aside from the extra code involved...

> That's what we can't afford to have. RAM is limited and we had to
> deal with millions of very big messages.

When you say 'deal with millions', what do you mean? That your rabbit 
needs to hold (in its queues) millions of 50Mb messages? If you fragment 
messages at 100k that would translate to 500 million messages. You'd 
need a fair amount of RAM for that since each message has some fixed 
memory footprint (in the small hundreds of bytes, iirc) even when paged 
to disk.

Regards,

Matthias.