[rabbitmq-discuss] Problem transferring large message

[Zabrane Mickael]

Hi Matthias,

> I have just tested twenty producers sending 50Mb messages to Rabbit as
> fast as they can. The combined rate was ~30Hz. And that's on a fairly modest machine. There were some long pauses when messages were getting paged to disk, and memory peaked at about 80% (so twice the configured threshold), but that's expected. No crashes.
> 
> With smaller messages the performance profile will be smoother, i.e. the pauses will be shorter and memory peaks will be lower. But there are downsides to fragmenting messages aside from the extra code involved...

Interesting.

>> That's what we can't afford to have. RAM is limited and we had to
>> deal with millions of very big messages.
> 
> When you say 'deal with millions', what do you mean? That your rabbit needs to hold (in its queues) millions of 50Mb messages? If you fragment messages at 100k that would translate to 500 million messages. You'd need a fair amount of RAM for that since each message has some fixed memory footprint (in the small hundreds of bytes, iirc) even when paged to disk.


Yep. Our system (large cluster) produce huge amount files on disk (0 to 5 millions per day) and we needed to analyse them at as fast as possible.
These messages stay a very short time on Rabbit queues. Splitting them helped us a lot.

Regards,
Zabrane